2024-05-06T13:21:20.314 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-06T13:21:20.315 DEBUG:teuthology.run:Teuthology command: teuthology --description orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} --name teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688245 --verbose --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688245/orig.config.yaml 2024-05-06T13:21:20.364 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-06T13:21:20.455 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688245 branch: squid description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} email: ceph-qa@ceph.com first_in_suite: false job_id: '7688245' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-only-match: - CEPHADM_ sha1: 70ce9539b62a15b9de429c1f0e91a1e012b099aa ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 70ce9539b62a15b9de429c1f0e91a1e012b099aa workunit: branch: squid sha1: 70ce9539b62a15b9de429c1f0e91a1e012b099aa owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 3735 sha1: 70ce9539b62a15b9de429c1f0e91a1e012b099aa sleep_before_teardown: 0 subset: 10/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_70ce9539b62a15b9de429c1f0e91a1e012b099aa/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 70ce9539b62a15b9de429c1f0e91a1e012b099aa targets: smithi077.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJohsaqWYVHYNRlz7jBCzGjRhYWqxUS1GR2fYpLwh5zEhXg0vrmTTjWDoEhZ2EuajqCBbSUQ0dkrh7VBd8HVGj4= smithi170.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDGW/aOJzVnJ8BaA7lVrBx7YOJewdcCurFHn9O6QQmS4wk7S61m/iZdGqVMkWBKU0u0DPwdrDlWuRNJ1JF/qd6M= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest false --force - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 180 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch ps - ceph versions - ceph -s - ceph orch ls - ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps --refresh - sleep 180 - ceph orch ps - ceph versions - ceph -s - ceph health detail - ceph versions | jq -e '.mgr | length == 2' - ceph mgr fail - sleep 180 - ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps --refresh - sleep 180 - ceph orch ps - ceph versions - ceph health detail - ceph -s - ceph mgr fail - sleep 180 - ceph orch ps - ceph versions - ceph -s - ceph health detail - ceph versions | jq -e '.mgr | length == 1' - ceph mgr fail - sleep 180 - ceph orch ps - ceph orch ls - ceph versions - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk '{print $2}') - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.mon | length == 2' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.y | awk '{print $2}') - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.mon | length == 1' - ceph versions | jq -e '.mon | keys' | grep $sha1 - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 5' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 2 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 7' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd --limit 1 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 8' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 1' - ceph versions | jq -e '.osd | keys' | grep $sha1 - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --services rgw.foo - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.rgw | length == 1' - ceph versions | jq -e '.rgw | keys' | grep $sha1 - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-03_21:08:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-06T13:21:20.455 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_70ce9539b62a15b9de429c1f0e91a1e012b099aa/qa; will attempt to use it 2024-05-06T13:21:20.456 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_70ce9539b62a15b9de429c1f0e91a1e012b099aa/qa/tasks 2024-05-06T13:21:20.456 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-06T13:21:20.458 INFO:teuthology.task.internal:Checking packages... 2024-05-06T13:21:20.485 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '70ce9539b62a15b9de429c1f0e91a1e012b099aa' 2024-05-06T13:21:20.485 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-06T13:21:20.485 INFO:teuthology.packaging:ref: None 2024-05-06T13:21:20.485 INFO:teuthology.packaging:tag: None 2024-05-06T13:21:20.485 INFO:teuthology.packaging:branch: squid 2024-05-06T13:21:20.485 INFO:teuthology.packaging:sha1: 70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:21:20.485 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-05-06T13:21:20.802 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2504.gb05eb275 2024-05-06T13:21:20.804 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-06T13:21:20.813 INFO:teuthology.task.internal:no buildpackages task found 2024-05-06T13:21:20.813 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-06T13:21:20.954 INFO:teuthology.task.internal:Saving configuration 2024-05-06T13:21:20.975 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-06T13:21:21.026 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-06T13:21:21.051 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688245', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-06 13:16:29.780457', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJohsaqWYVHYNRlz7jBCzGjRhYWqxUS1GR2fYpLwh5zEhXg0vrmTTjWDoEhZ2EuajqCBbSUQ0dkrh7VBd8HVGj4='} 2024-05-06T13:21:21.077 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688245', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-06 13:16:29.779135', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDGW/aOJzVnJ8BaA7lVrBx7YOJewdcCurFHn9O6QQmS4wk7S61m/iZdGqVMkWBKU0u0DPwdrDlWuRNJ1JF/qd6M='} 2024-05-06T13:21:21.077 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-06T13:21:21.088 INFO:teuthology.task.internal:roles: ubuntu@smithi077.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-05-06T13:21:21.088 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-05-06T13:21:21.088 INFO:teuthology.run_tasks:Running task console_log... 2024-05-06T13:21:21.213 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1a9f1240d0>, signals=[15]) 2024-05-06T13:21:21.213 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-06T13:21:21.224 INFO:teuthology.task.internal:Opening connections... 2024-05-06T13:21:21.224 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi077.front.sepia.ceph.com 2024-05-06T13:21:21.227 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T13:21:21.309 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi170.front.sepia.ceph.com 2024-05-06T13:21:21.310 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T13:21:21.390 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-06T13:21:21.399 DEBUG:teuthology.orchestra.run.smithi077:> uname -m 2024-05-06T13:21:21.417 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64 2024-05-06T13:21:21.417 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release 2024-05-06T13:21:21.475 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="CentOS Stream" 2024-05-06T13:21:21.475 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="9" 2024-05-06T13:21:21.475 INFO:teuthology.orchestra.run.smithi077.stdout:ID="centos" 2024-05-06T13:21:21.475 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE="rhel fedora" 2024-05-06T13:21:21.475 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="9" 2024-05-06T13:21:21.475 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM_ID="platform:el9" 2024-05-06T13:21:21.475 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-06T13:21:21.475 INFO:teuthology.orchestra.run.smithi077.stdout:ANSI_COLOR="0;31" 2024-05-06T13:21:21.475 INFO:teuthology.orchestra.run.smithi077.stdout:LOGO="fedora-logo-icon" 2024-05-06T13:21:21.475 INFO:teuthology.orchestra.run.smithi077.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-06T13:21:21.476 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://centos.org/" 2024-05-06T13:21:21.476 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-06T13:21:21.476 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-06T13:21:21.476 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-06T13:21:21.476 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2024-05-06T13:21:21.496 DEBUG:teuthology.orchestra.run.smithi170:> uname -m 2024-05-06T13:21:21.515 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64 2024-05-06T13:21:21.515 DEBUG:teuthology.orchestra.run.smithi170:> cat /etc/os-release 2024-05-06T13:21:21.573 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="CentOS Stream" 2024-05-06T13:21:21.573 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="9" 2024-05-06T13:21:21.573 INFO:teuthology.orchestra.run.smithi170.stdout:ID="centos" 2024-05-06T13:21:21.573 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="rhel fedora" 2024-05-06T13:21:21.573 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="9" 2024-05-06T13:21:21.573 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el9" 2024-05-06T13:21:21.574 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-06T13:21:21.574 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31" 2024-05-06T13:21:21.574 INFO:teuthology.orchestra.run.smithi170.stdout:LOGO="fedora-logo-icon" 2024-05-06T13:21:21.574 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-06T13:21:21.574 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://centos.org/" 2024-05-06T13:21:21.574 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-06T13:21:21.574 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-06T13:21:21.574 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-06T13:21:21.574 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2024-05-06T13:21:21.595 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-06T13:21:21.608 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-06T13:21:21.652 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-06T13:21:21.653 DEBUG:teuthology.orchestra.run.smithi077:> test '!' -e /home/ubuntu/cephtest 2024-05-06T13:21:21.655 DEBUG:teuthology.orchestra.run.smithi170:> test '!' -e /home/ubuntu/cephtest 2024-05-06T13:21:21.672 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-06T13:21:21.724 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-06T13:21:21.724 DEBUG:teuthology.orchestra.run.smithi077:> test -z $(ls -A /var/lib/ceph) 2024-05-06T13:21:21.726 DEBUG:teuthology.orchestra.run.smithi170:> test -z $(ls -A /var/lib/ceph) 2024-05-06T13:21:21.748 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-06T13:21:21.819 INFO:teuthology.run_tasks:Running task kernel... 2024-05-06T13:21:21.837 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-05-06T13:21:21.837 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-05-06T13:21:21.838 DEBUG:teuthology.orchestra.run.smithi077:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-06T13:21:21.838 DEBUG:teuthology.orchestra.run.smithi170:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-06T13:21:21.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T13:21:21.855 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2024-05-06T13:21:21.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T13:21:21.856 DEBUG:teuthology.orchestra.run.smithi170:> uname -r 2024-05-06T13:21:21.912 INFO:teuthology.orchestra.run.smithi077.stdout:5.14.0-437.el9.x86_64 2024-05-06T13:21:21.912 INFO:teuthology.task.kernel:Running kernel on smithi077: 5.14.0-437.el9.x86_64 2024-05-06T13:21:21.912 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum install -y kernel 2024-05-06T13:21:21.913 INFO:teuthology.orchestra.run.smithi170.stdout:5.14.0-437.el9.x86_64 2024-05-06T13:21:21.913 INFO:teuthology.task.kernel:Running kernel on smithi170: 5.14.0-437.el9.x86_64 2024-05-06T13:21:21.914 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum install -y kernel 2024-05-06T13:21:24.492 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - BaseOS 9.8 MB/s | 8.0 MB 00:00 2024-05-06T13:21:24.749 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - BaseOS 6.7 MB/s | 8.0 MB 00:01 2024-05-06T13:21:26.393 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-05-06T13:21:26.542 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-05-06T13:21:31.785 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.2 MB 00:01 2024-05-06T13:21:31.826 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 6.2 MB 00:01 2024-05-06T13:21:33.235 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 16 kB 00:00 2024-05-06T13:21:33.277 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - Extras packages 98 kB/s | 16 kB 00:00 2024-05-06T13:21:34.588 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-05-06T13:21:34.945 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-05-06T13:21:39.711 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-05-06T13:21:40.087 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-05-06T13:21:41.576 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-06T13:21:41.577 INFO:teuthology.orchestra.run.smithi170.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-06T13:21:41.680 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-05-06T13:21:41.681 INFO:teuthology.orchestra.run.smithi170.stdout:Nothing to do. 2024-05-06T13:21:41.681 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-05-06T13:21:41.805 DEBUG:teuthology.orchestra.run.smithi170:> echo no | sudo yum reinstall kernel || true 2024-05-06T13:21:41.960 INFO:teuthology.orchestra.run.smithi077.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-06T13:21:41.960 INFO:teuthology.orchestra.run.smithi077.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-06T13:21:42.049 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-05-06T13:21:42.050 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2024-05-06T13:21:42.050 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-05-06T13:21:42.152 DEBUG:teuthology.orchestra.run.smithi077:> echo no | sudo yum reinstall kernel || true 2024-05-06T13:21:42.300 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:03 ago on Mon 06 May 2024 01:21:39 PM UTC. 2024-05-06T13:21:42.443 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-05-06T13:21:42.451 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-05-06T13:21:42.451 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2024-05-06T13:21:42.451 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-05-06T13:21:42.451 INFO:teuthology.orchestra.run.smithi170.stdout:Reinstalling: 2024-05-06T13:21:42.451 INFO:teuthology.orchestra.run.smithi170.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-06T13:21:42.451 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-05-06T13:21:42.451 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-05-06T13:21:42.451 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-05-06T13:21:42.451 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-05-06T13:21:42.452 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 255 k 2024-05-06T13:21:42.452 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 0 2024-05-06T13:21:42.452 INFO:teuthology.orchestra.run.smithi170.stderr:Operation aborted. 2024-05-06T13:21:42.511 INFO:teuthology.orchestra.run.smithi170.stdout:Is this ok [y/N]: 2024-05-06T13:21:42.511 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum reinstall -y kernel || true 2024-05-06T13:21:42.650 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:00:02 ago on Mon 06 May 2024 01:21:40 PM UTC. 2024-05-06T13:21:42.779 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-05-06T13:21:42.787 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-05-06T13:21:42.787 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2024-05-06T13:21:42.787 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-05-06T13:21:42.788 INFO:teuthology.orchestra.run.smithi077.stdout:Reinstalling: 2024-05-06T13:21:42.788 INFO:teuthology.orchestra.run.smithi077.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-06T13:21:42.788 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:21:42.788 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-05-06T13:21:42.788 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-05-06T13:21:42.788 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:21:42.788 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 255 k 2024-05-06T13:21:42.788 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 0 2024-05-06T13:21:42.788 INFO:teuthology.orchestra.run.smithi077.stderr:Operation aborted. 2024-05-06T13:21:42.843 INFO:teuthology.orchestra.run.smithi077.stdout:Is this ok [y/N]: 2024-05-06T13:21:42.843 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum reinstall -y kernel || true 2024-05-06T13:21:43.005 INFO:teuthology.orchestra.run.smithi170.stdout:Last metadata expiration check: 0:00:04 ago on Mon 06 May 2024 01:21:39 PM UTC. 2024-05-06T13:21:43.121 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-05-06T13:21:43.122 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-05-06T13:21:43.122 INFO:teuthology.orchestra.run.smithi170.stdout: Package Architecture Version Repository Size 2024-05-06T13:21:43.122 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-05-06T13:21:43.122 INFO:teuthology.orchestra.run.smithi170.stdout:Reinstalling: 2024-05-06T13:21:43.122 INFO:teuthology.orchestra.run.smithi170.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-06T13:21:43.122 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-05-06T13:21:43.122 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-05-06T13:21:43.122 INFO:teuthology.orchestra.run.smithi170.stdout:================================================================================ 2024-05-06T13:21:43.122 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-05-06T13:21:43.122 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 255 k 2024-05-06T13:21:43.123 INFO:teuthology.orchestra.run.smithi170.stdout:Installed size: 0 2024-05-06T13:21:43.123 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-05-06T13:21:43.335 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:00:03 ago on Mon 06 May 2024 01:21:40 PM UTC. 2024-05-06T13:21:43.451 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-05-06T13:21:43.452 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-05-06T13:21:43.452 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2024-05-06T13:21:43.452 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-05-06T13:21:43.452 INFO:teuthology.orchestra.run.smithi077.stdout:Reinstalling: 2024-05-06T13:21:43.453 INFO:teuthology.orchestra.run.smithi077.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-06T13:21:43.453 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:21:43.453 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-05-06T13:21:43.453 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-05-06T13:21:43.453 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:21:43.453 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 255 k 2024-05-06T13:21:43.453 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 0 2024-05-06T13:21:43.453 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-05-06T13:21:43.580 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-5.14.0-437.el9.x86_64.rpm 960 kB/s | 255 kB 00:00 2024-05-06T13:21:43.580 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-05-06T13:21:43.581 INFO:teuthology.orchestra.run.smithi170.stdout:Total 556 kB/s | 255 kB 00:00 2024-05-06T13:21:43.598 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-05-06T13:21:43.787 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-05-06T13:21:43.788 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-05-06T13:21:43.792 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-05-06T13:21:43.793 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-05-06T13:21:43.881 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.6 MB/s | 255 kB 00:00 2024-05-06T13:21:43.881 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-05-06T13:21:43.881 INFO:teuthology.orchestra.run.smithi077.stdout:Total 594 kB/s | 255 kB 00:00 2024-05-06T13:21:43.897 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-05-06T13:21:43.916 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-05-06T13:21:43.982 INFO:teuthology.orchestra.run.smithi170.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-06T13:21:43.982 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-06T13:21:44.088 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-05-06T13:21:44.088 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-05-06T13:21:44.092 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-05-06T13:21:44.093 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-05-06T13:21:44.220 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-05-06T13:21:44.299 INFO:teuthology.orchestra.run.smithi077.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-06T13:21:44.299 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-06T13:21:44.657 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-05-06T13:21:44.657 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-06T13:21:45.066 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-05-06T13:21:45.066 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-06T13:21:45.374 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-06T13:21:45.374 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-05-06T13:21:45.374 INFO:teuthology.orchestra.run.smithi170.stdout:Reinstalled: 2024-05-06T13:21:45.374 INFO:teuthology.orchestra.run.smithi170.stdout: kernel-5.14.0-437.el9.x86_64 2024-05-06T13:21:45.374 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-05-06T13:21:45.374 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-05-06T13:21:45.551 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q kernel | sort -rV | head -n 1 2024-05-06T13:21:45.598 INFO:teuthology.orchestra.run.smithi170.stdout:kernel-5.14.0-437.el9.x86_64 2024-05-06T13:21:45.599 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-05-06T13:21:45.599 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-05-06T13:21:45.599 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-05-06T13:21:45.599 DEBUG:teuthology.orchestra.run.smithi170:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-06T13:21:45.628 INFO:teuthology.orchestra.run.smithi170.stdout:ttyS1 2024-05-06T13:21:45.652 DEBUG:teuthology.parallel:result is None 2024-05-06T13:21:45.752 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-06T13:21:45.752 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:21:45.752 INFO:teuthology.orchestra.run.smithi077.stdout:Reinstalled: 2024-05-06T13:21:45.753 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-5.14.0-437.el9.x86_64 2024-05-06T13:21:45.753 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:21:45.753 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-05-06T13:21:45.911 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q kernel | sort -rV | head -n 1 2024-05-06T13:21:45.958 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-5.14.0-437.el9.x86_64 2024-05-06T13:21:45.958 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-05-06T13:21:45.959 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-05-06T13:21:45.959 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-06T13:21:45.959 DEBUG:teuthology.orchestra.run.smithi077:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-06T13:21:46.000 INFO:teuthology.orchestra.run.smithi077.stdout:ttyS1 2024-05-06T13:21:46.017 DEBUG:teuthology.parallel:result is None 2024-05-06T13:21:46.018 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-06T13:21:46.027 INFO:teuthology.task.internal:Creating test directory... 2024-05-06T13:21:46.028 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-06T13:21:46.060 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-06T13:21:46.075 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-06T13:21:46.082 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-06T13:21:46.093 INFO:teuthology.task.internal:Creating archive directory... 2024-05-06T13:21:46.093 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-06T13:21:46.116 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-06T13:21:46.143 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-06T13:21:46.150 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-06T13:21:46.150 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-06T13:21:46.178 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-06T13:21:46.206 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T13:21:46.210 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T13:21:46.215 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T13:21:46.219 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-06T13:21:46.221 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-06T13:21:46.228 INFO:teuthology.task.internal:Configuring sudo... 2024-05-06T13:21:46.228 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-06T13:21:46.259 DEBUG:teuthology.orchestra.run.smithi170:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-06T13:21:46.285 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-06T13:21:46.295 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-06T13:21:46.296 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-06T13:21:46.324 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-06T13:21:46.340 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-06T13:21:46.416 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-06T13:21:46.460 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-06T13:21:46.541 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-06T13:21:46.580 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-05-06T13:21:46.580 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-06T13:21:46.644 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-06T13:21:46.683 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-06T13:21:46.729 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-06T13:21:46.811 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-06T13:21:46.850 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-05-06T13:21:46.850 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-06T13:21:46.914 DEBUG:teuthology.orchestra.run.smithi077:> sudo service rsyslog restart 2024-05-06T13:21:46.917 DEBUG:teuthology.orchestra.run.smithi170:> sudo service rsyslog restart 2024-05-06T13:21:46.959 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-06T13:21:46.992 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-06T13:21:47.358 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-06T13:21:47.366 INFO:teuthology.task.internal:Starting timer... 2024-05-06T13:21:47.366 INFO:teuthology.run_tasks:Running task pcp... 2024-05-06T13:21:47.403 INFO:teuthology.run_tasks:Running task selinux... 2024-05-06T13:21:47.443 DEBUG:teuthology.orchestra.run.smithi077:> sudo service auditd rotate 2024-05-06T13:21:47.492 INFO:teuthology.orchestra.run.smithi077.stdout:Rotating logs: 2024-05-06T13:21:47.494 DEBUG:teuthology.orchestra.run.smithi170:> sudo service auditd rotate 2024-05-06T13:21:47.566 INFO:teuthology.orchestra.run.smithi170.stdout:Rotating logs: 2024-05-06T13:21:47.567 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-06T13:21:47.567 DEBUG:teuthology.orchestra.run.smithi077:> /usr/sbin/getenforce 2024-05-06T13:21:47.592 INFO:teuthology.orchestra.run.smithi077.stdout:Permissive 2024-05-06T13:21:47.592 DEBUG:teuthology.orchestra.run.smithi170:> /usr/sbin/getenforce 2024-05-06T13:21:47.613 INFO:teuthology.orchestra.run.smithi170.stdout:Permissive 2024-05-06T13:21:47.613 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi077.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi170.front.sepia.ceph.com': 'permissive'} 2024-05-06T13:21:47.613 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-06T13:21:47.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T13:21:47.657 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-06T13:21:47.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T13:21:47.683 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-06T13:21:47.683 DEBUG:teuthology.orchestra.run.smithi077:> sudo /usr/sbin/setenforce permissive 2024-05-06T13:21:47.722 DEBUG:teuthology.orchestra.run.smithi170:> sudo /usr/sbin/setenforce permissive 2024-05-06T13:21:47.757 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-06T13:21:47.767 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-06T13:21:47.785 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-06T13:21:47.786 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi077.front.sepia.ceph.com,smithi170.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-06T13:25:37.982 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi077.front.sepia.ceph.com'), Remote(name='ubuntu@smithi170.front.sepia.ceph.com')] 2024-05-06T13:25:37.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-05-06T13:25:37.985 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T13:25:38.062 DEBUG:teuthology.orchestra.run.smithi077:> true 2024-05-06T13:25:38.136 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-05-06T13:25:38.137 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-05-06T13:25:38.137 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-06T13:25:38.212 DEBUG:teuthology.orchestra.run.smithi170:> true 2024-05-06T13:25:38.296 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-05-06T13:25:38.297 INFO:teuthology.run_tasks:Running task clock... 2024-05-06T13:25:38.307 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-06T13:25:38.307 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-06T13:25:38.308 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-06T13:25:38.310 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-06T13:25:38.310 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-06T13:25:38.334 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-06T13:25:38.347 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-06T13:25:38.363 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-06T13:25:38.369 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: ntpd: command not found 2024-05-06T13:25:38.376 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-06T13:25:38.379 INFO:teuthology.orchestra.run.smithi077.stdout:506 Cannot talk to daemon 2024-05-06T13:25:38.392 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-06T13:25:38.398 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: ntpd: command not found 2024-05-06T13:25:38.405 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-06T13:25:38.408 INFO:teuthology.orchestra.run.smithi170.stdout:506 Cannot talk to daemon 2024-05-06T13:25:38.421 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-06T13:25:38.434 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-06T13:25:38.468 INFO:teuthology.orchestra.run.smithi077.stderr:bash: line 1: ntpq: command not found 2024-05-06T13:25:38.471 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-06T13:25:38.472 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-05-06T13:25:38.472 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T13:25:38.472 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T13:25:38.472 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T13:25:38.472 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T13:25:38.491 INFO:teuthology.orchestra.run.smithi170.stderr:bash: line 1: ntpq: command not found 2024-05-06T13:25:38.494 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-06T13:25:38.494 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2024-05-06T13:25:38.495 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T13:25:38.495 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T13:25:38.495 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T13:25:38.495 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T13:25:38.495 INFO:teuthology.run_tasks:Running task cephadm... 2024-05-06T13:25:38.596 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': '70ce9539b62a15b9de429c1f0e91a1e012b099aa'} 2024-05-06T13:25:38.596 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-05-06T13:25:38.597 INFO:tasks.cephadm:Cluster fsid is 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:25:38.597 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-05-06T13:25:38.597 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.77', 'mon.c': '[v2:172.21.15.77:3301,v1:172.21.15.77:6790]', 'mon.b': '172.21.15.170'} 2024-05-06T13:25:38.597 INFO:tasks.cephadm:First mon is mon.a on smithi077 2024-05-06T13:25:38.598 INFO:tasks.cephadm:First mgr is y 2024-05-06T13:25:38.598 INFO:tasks.cephadm:Normalizing hostnames... 2024-05-06T13:25:38.598 DEBUG:teuthology.orchestra.run.smithi077:> sudo hostname $(hostname -s) 2024-05-06T13:25:38.623 DEBUG:teuthology.orchestra.run.smithi170:> sudo hostname $(hostname -s) 2024-05-06T13:25:38.647 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-05-06T13:25:38.647 DEBUG:teuthology.orchestra.run.smithi077:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-06T13:25:38.903 INFO:teuthology.orchestra.run.smithi077.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 May 6 13:25 /home/ubuntu/cephtest/cephadm 2024-05-06T13:25:38.903 DEBUG:teuthology.orchestra.run.smithi170:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-06T13:25:39.082 INFO:teuthology.orchestra.run.smithi170.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 May 6 13:25 /home/ubuntu/cephtest/cephadm 2024-05-06T13:25:39.082 DEBUG:teuthology.orchestra.run.smithi077:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-06T13:25:39.098 DEBUG:teuthology.orchestra.run.smithi170:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-06T13:25:39.129 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-05-06T13:25:39.130 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-05-06T13:25:39.140 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-05-06T13:25:40.599 INFO:teuthology.orchestra.run.smithi077.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-05-06T13:25:40.721 INFO:teuthology.orchestra.run.smithi170.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-05-06T13:26:09.540 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-05-06T13:26:09.540 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-05-06T13:26:09.540 INFO:teuthology.orchestra.run.smithi170.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-05-06T13:26:09.540 INFO:teuthology.orchestra.run.smithi170.stdout: "repo_digests": [ 2024-05-06T13:26:09.540 INFO:teuthology.orchestra.run.smithi170.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-05-06T13:26:09.540 INFO:teuthology.orchestra.run.smithi170.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-05-06T13:26:09.540 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-05-06T13:26:09.541 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-05-06T13:26:18.424 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:26:18.424 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-05-06T13:26:18.424 INFO:teuthology.orchestra.run.smithi077.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-05-06T13:26:18.424 INFO:teuthology.orchestra.run.smithi077.stdout: "repo_digests": [ 2024-05-06T13:26:18.424 INFO:teuthology.orchestra.run.smithi077.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-05-06T13:26:18.424 INFO:teuthology.orchestra.run.smithi077.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-05-06T13:26:18.424 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-05-06T13:26:18.424 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:26:18.442 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph 2024-05-06T13:26:18.500 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /etc/ceph 2024-05-06T13:26:18.535 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 777 /etc/ceph 2024-05-06T13:26:18.558 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 777 /etc/ceph 2024-05-06T13:26:18.603 INFO:tasks.cephadm:Writing seed config... 2024-05-06T13:26:18.604 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-05-06T13:26:18.604 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-05-06T13:26:18.604 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-05-06T13:26:18.604 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-05-06T13:26:18.604 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-05-06T13:26:18.605 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-05-06T13:26:18.605 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-05-06T13:26:18.605 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-05-06T13:26:18.605 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-05-06T13:26:18.605 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-05-06T13:26:18.605 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-05-06T13:26:18.620 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 20e59cb4-0bac-11ef-bc97-c7b262605968 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-05-06T13:26:18.621 DEBUG:teuthology.orchestra.run.smithi077:mon.a> sudo journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mon.a.service 2024-05-06T13:26:18.663 DEBUG:teuthology.orchestra.run.smithi077:mgr.y> sudo journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mgr.y.service 2024-05-06T13:26:18.706 INFO:tasks.cephadm:Bootstrapping... 2024-05-06T13:26:18.706 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.77 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-05-06T13:26:18.895 INFO:teuthology.orchestra.run.smithi077.stderr:-------------------------------------------------------------------------------- 2024-05-06T13:26:18.896 INFO:teuthology.orchestra.run.smithi077.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '20e59cb4-0bac-11ef-bc97-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.77', '--skip-admin-label'] 2024-05-06T13:26:18.920 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: 4.9.4-dev 2024-05-06T13:26:18.922 INFO:teuthology.orchestra.run.smithi077.stderr:Verifying podman|docker is present... 2024-05-06T13:26:18.943 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: 4.9.4-dev 2024-05-06T13:26:18.945 INFO:teuthology.orchestra.run.smithi077.stderr:Verifying lvm2 is present... 2024-05-06T13:26:18.945 INFO:teuthology.orchestra.run.smithi077.stderr:Verifying time synchronization is in place... 2024-05-06T13:26:18.952 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-05-06T13:26:18.958 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: inactive 2024-05-06T13:26:18.964 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: enabled 2024-05-06T13:26:18.970 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: active 2024-05-06T13:26:18.971 INFO:teuthology.orchestra.run.smithi077.stderr:Unit chronyd.service is enabled and running 2024-05-06T13:26:18.971 INFO:teuthology.orchestra.run.smithi077.stderr:Repeating the final host check... 2024-05-06T13:26:18.993 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: 4.9.4-dev 2024-05-06T13:26:18.995 INFO:teuthology.orchestra.run.smithi077.stderr:podman (/bin/podman) version 4.9.4 is present 2024-05-06T13:26:18.995 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl is present 2024-05-06T13:26:18.995 INFO:teuthology.orchestra.run.smithi077.stderr:lvcreate is present 2024-05-06T13:26:19.000 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-05-06T13:26:19.006 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: inactive 2024-05-06T13:26:19.013 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: enabled 2024-05-06T13:26:19.019 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: active 2024-05-06T13:26:19.020 INFO:teuthology.orchestra.run.smithi077.stderr:Unit chronyd.service is enabled and running 2024-05-06T13:26:19.020 INFO:teuthology.orchestra.run.smithi077.stderr:Host looks OK 2024-05-06T13:26:19.020 INFO:teuthology.orchestra.run.smithi077.stderr:Cluster fsid: 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:26:19.020 INFO:teuthology.orchestra.run.smithi077.stderr:Acquiring lock 140510811422576 on /run/cephadm/20e59cb4-0bac-11ef-bc97-c7b262605968.lock 2024-05-06T13:26:19.020 INFO:teuthology.orchestra.run.smithi077.stderr:Lock 140510811422576 acquired on /run/cephadm/20e59cb4-0bac-11ef-bc97-c7b262605968.lock 2024-05-06T13:26:19.020 INFO:teuthology.orchestra.run.smithi077.stderr:Verifying IP 172.21.15.77 port 3300 ... 2024-05-06T13:26:19.021 INFO:teuthology.orchestra.run.smithi077.stderr:Verifying IP 172.21.15.77 port 6789 ... 2024-05-06T13:26:19.021 INFO:teuthology.orchestra.run.smithi077.stderr:Base mon IP is 172.21.15.77, final addrv is [v2:172.21.15.77:3300,v1:172.21.15.77:6789] 2024-05-06T13:26:19.023 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.77 metric 100 2024-05-06T13:26:19.023 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.77 metric 100 2024-05-06T13:26:19.026 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-05-06T13:26:19.026 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-05-06T13:26:19.026 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-05-06T13:26:19.028 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-05-06T13:26:19.028 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-05-06T13:26:19.028 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-05-06T13:26:19.028 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-05-06T13:26:19.028 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: inet6 fe80::dcdd:ce8d:9b2e:3047/64 scope link noprefixroute 2024-05-06T13:26:19.028 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-05-06T13:26:19.029 INFO:teuthology.orchestra.run.smithi077.stderr:Mon IP `172.21.15.77` is in CIDR network `172.21.0.0/20` 2024-05-06T13:26:19.029 INFO:teuthology.orchestra.run.smithi077.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-05-06T13:26:19.030 INFO:teuthology.orchestra.run.smithi077.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-05-06T13:26:19.132 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-05-06T13:26:20.405 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: Getting image source signatures 2024-05-06T13:26:20.405 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-05-06T13:26:20.405 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-05-06T13:26:20.406 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-05-06T13:26:20.406 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-05-06T13:26:20.406 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-05-06T13:26:20.406 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-05-06T13:26:20.536 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: Writing manifest to image destination 2024-05-06T13:26:20.872 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-05-06T13:26:21.274 INFO:teuthology.orchestra.run.smithi077.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-05-06T13:26:21.821 INFO:teuthology.orchestra.run.smithi077.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-05-06T13:26:21.821 INFO:teuthology.orchestra.run.smithi077.stderr:Extracting ceph user uid/gid from container image... 2024-05-06T13:26:22.230 INFO:teuthology.orchestra.run.smithi077.stderr:stat: 167 167 2024-05-06T13:26:22.719 INFO:teuthology.orchestra.run.smithi077.stderr:Creating initial keys... 2024-05-06T13:26:23.110 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-authtool: AQB/2jhm4KOSARAAHxiOqFID5TktOsY75EndPA== 2024-05-06T13:26:27.241 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-authtool: AQCC2jhmE5v6IRAAEUOoVROTNrqqp2sa25Eq4A== 2024-05-06T13:26:33.664 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-authtool: AQCJ2jhmqka1IhAA4m5brD4zt0cNd1/5DrK8HA== 2024-05-06T13:26:34.305 INFO:teuthology.orchestra.run.smithi077.stderr:Creating initial monmap... 2024-05-06T13:26:34.712 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-06T13:26:34.713 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-05-06T13:26:34.713 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:26:34.713 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-06T13:26:35.235 INFO:teuthology.orchestra.run.smithi077.stderr:monmaptool for a [v2:172.21.15.77:3300,v1:172.21.15.77:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-06T13:26:35.235 INFO:teuthology.orchestra.run.smithi077.stderr:setting min_mon_release = octopus 2024-05-06T13:26:35.236 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/monmaptool: set fsid to 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:26:35.236 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-06T13:26:35.236 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-05-06T13:26:35.236 INFO:teuthology.orchestra.run.smithi077.stderr:Creating mon... 2024-05-06T13:26:36.057 INFO:teuthology.orchestra.run.smithi077.stderr:create mon.a on 2024-05-06T13:26:36.255 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-05-06T13:26:36.402 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-20e59cb4-0bac-11ef-bc97-c7b262605968.target → /etc/systemd/system/ceph-20e59cb4-0bac-11ef-bc97-c7b262605968.target. 2024-05-06T13:26:36.402 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-20e59cb4-0bac-11ef-bc97-c7b262605968.target → /etc/systemd/system/ceph-20e59cb4-0bac-11ef-bc97-c7b262605968.target. 2024-05-06T13:26:36.717 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Failed to reset failed state of unit ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mon.a.service: Unit ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mon.a.service not loaded. 2024-05-06T13:26:36.724 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Created symlink /etc/systemd/system/ceph-20e59cb4-0bac-11ef-bc97-c7b262605968.target.wants/ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mon.a.service → /etc/systemd/system/ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@.service. 2024-05-06T13:26:37.421 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: disabled 2024-05-06T13:26:37.428 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: inactive 2024-05-06T13:26:37.428 INFO:teuthology.orchestra.run.smithi077.stderr:firewalld.service is not enabled 2024-05-06T13:26:37.428 INFO:teuthology.orchestra.run.smithi077.stderr:Not possible to enable service . firewalld.service is not available 2024-05-06T13:26:37.429 INFO:teuthology.orchestra.run.smithi077.stderr:Waiting for mon to start... 2024-05-06T13:26:37.429 INFO:teuthology.orchestra.run.smithi077.stderr:Waiting for mon... 2024-05-06T13:26:37.476 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:37 smithi077 systemd[1]: Started Ceph mon.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:26:37.951 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: cluster: 2024-05-06T13:26:37.951 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: id: 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:26:37.951 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-05-06T13:26:37.952 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-05-06T13:26:37.952 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: services: 2024-05-06T13:26:37.952 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.560843s) 2024-05-06T13:26:37.952 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mgr: no daemons active 2024-05-06T13:26:37.952 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-05-06T13:26:37.952 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-05-06T13:26:37.952 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: data: 2024-05-06T13:26:37.952 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-05-06T13:26:37.952 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-05-06T13:26:37.952 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-05-06T13:26:37.953 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: pgs: 2024-05-06T13:26:37.953 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-05-06T13:26:38.801 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:38 smithi077 ceph-mon[22078]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-06T13:26:38.801 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:38 smithi077 ceph-mon[22078]: monmap e1: 1 mons at {a=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0]} 2024-05-06T13:26:38.801 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:38 smithi077 ceph-mon[22078]: fsmap 2024-05-06T13:26:38.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:38 smithi077 ceph-mon[22078]: osdmap e1: 0 total, 0 up, 0 in 2024-05-06T13:26:38.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:38 smithi077 ceph-mon[22078]: mgrmap e1: no daemons active 2024-05-06T13:26:38.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:38 smithi077 ceph-mon[22078]: from='client.? 172.21.15.77:0/4170924274' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-06T13:26:40.780 INFO:teuthology.orchestra.run.smithi077.stderr:mon is available 2024-05-06T13:26:40.780 INFO:teuthology.orchestra.run.smithi077.stderr:Assimilating anything we can from ceph.conf... 2024-05-06T13:26:41.264 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-05-06T13:26:41.265 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: [global] 2024-05-06T13:26:41.265 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: fsid = 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:26:41.265 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.77:3300,v1:172.21.15.77:6789] 2024-05-06T13:26:41.265 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-05-06T13:26:41.265 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-05-06T13:26:41.265 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-05-06T13:26:41.265 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-05-06T13:26:41.265 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-05-06T13:26:41.265 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: [mgr] 2024-05-06T13:26:41.265 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-05-06T13:26:41.265 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-05-06T13:26:41.266 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-05-06T13:26:41.266 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: [osd] 2024-05-06T13:26:41.266 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-05-06T13:26:41.266 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-05-06T13:26:41.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:41 smithi077 ceph-mon[22078]: from='client.? 172.21.15.77:0/3249644945' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-06T13:26:41.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:41 smithi077 ceph-mon[22078]: from='client.? 172.21.15.77:0/3249644945' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-06T13:26:41.852 INFO:teuthology.orchestra.run.smithi077.stderr:Generating new minimal ceph.conf... 2024-05-06T13:26:42.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:42 smithi077 ceph-mon[22078]: from='client.? 172.21.15.77:0/1762607703' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:26:42.901 INFO:teuthology.orchestra.run.smithi077.stderr:Restarting the monitor... 2024-05-06T13:26:43.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:42 smithi077 systemd[1]: Stopping Ceph mon.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:26:43.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:42 smithi077 bash[22300]: Error: no container with name or ID "ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon.a" found: no such container 2024-05-06T13:26:43.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-a[22074]: 2024-05-06T13:26:43.075+0000 7f0914feb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T13:26:43.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-a[22074]: 2024-05-06T13:26:43.075+0000 7f0914feb700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-06T13:26:43.592 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:43 smithi077 podman[22313]: 2024-05-06 13:26:43.322561297 +0000 UTC m=+0.357657968 container died 90c46901d3863b37db931c4b7a2881625a49c29d43b8647fe3d96cd9f1e3e245 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-a, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, version=8, io.buildah.version=1.19.8, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, ceph=True, vendor=Red Hat, Inc., name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-05-06T13:26:43.942 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:43 smithi077 podman[22313]: 2024-05-06 13:26:43.67609713 +0000 UTC m=+0.711193795 container cleanup 90c46901d3863b37db931c4b7a2881625a49c29d43b8647fe3d96cd9f1e3e245 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, RELEASE=HEAD, vcs-type=git, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, maintainer=Guillaume Abrioux , distribution-scope=public, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream) 2024-05-06T13:26:43.943 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:43 smithi077 bash[22313]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-a 2024-05-06T13:26:43.943 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:43 smithi077 bash[22341]: Error: no container with name or ID "ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon.a" found: no such container 2024-05-06T13:26:44.197 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:43 smithi077 podman[22328]: 2024-05-06 13:26:43.943745357 +0000 UTC m=+0.620112991 container remove 90c46901d3863b37db931c4b7a2881625a49c29d43b8647fe3d96cd9f1e3e245 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, name=centos-stream, com.redhat.component=centos-stream-container, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream) 2024-05-06T13:26:44.197 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:43 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-06T13:26:44.197 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:43 smithi077 systemd[1]: Stopped Ceph mon.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:26:44.197 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:43 smithi077 systemd[1]: Starting Ceph mon.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:26:44.198 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 podman[22409]: 2024-05-06 13:26:44.186334074 +0000 UTC m=+0.116154757 container create ac46f91694f871a4c86d9233330db0effb6a0b7cdd415dbb8c13ed18d2c6e0a7 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-a, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, release=754, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, name=centos-stream, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, RELEASE=HEAD, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git) 2024-05-06T13:26:44.198 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 podman[22409]: 2024-05-06 13:26:44.098944566 +0000 UTC m=+0.028765204 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-05-06T13:26:44.441 INFO:teuthology.orchestra.run.smithi077.stderr:Setting mon public_network to 172.21.0.0/20 2024-05-06T13:26:44.466 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 podman[22409]: 2024-05-06 13:26:44.380118397 +0000 UTC m=+0.309939035 container init ac46f91694f871a4c86d9233330db0effb6a0b7cdd415dbb8c13ed18d2c6e0a7 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-a, io.buildah.version=1.19.8, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, GIT_BRANCH=HEAD, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, distribution-scope=public, com.redhat.component=centos-stream-container, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, RELEASE=HEAD) 2024-05-06T13:26:44.467 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 podman[22409]: 2024-05-06 13:26:44.382945526 +0000 UTC m=+0.312766164 container start ac46f91694f871a4c86d9233330db0effb6a0b7cdd415dbb8c13ed18d2c6e0a7 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-a, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, release=754, GIT_BRANCH=HEAD, architecture=x86_64, com.redhat.component=centos-stream-container, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, vendor=Red Hat, Inc., GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870) 2024-05-06T13:26:44.467 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: set uid:gid to 167:167 (ceph:ceph) 2024-05-06T13:26:44.467 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-05-06T13:26:44.467 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: pidfile_write: ignore empty --pid-file 2024-05-06T13:26:44.467 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: load: jerasure load: lrc 2024-05-06T13:26:44.467 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: RocksDB version: 6.15.5 2024-05-06T13:26:44.467 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-05-06T13:26:44.467 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Compile date Apr 18 2022 2024-05-06T13:26:44.467 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: DB SUMMARY 2024-05-06T13:26:44.467 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: DB Session ID: GE7NOLOHLYPLH2NUSB97 2024-05-06T13:26:44.467 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: CURRENT file: CURRENT 2024-05-06T13:26:44.467 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: IDENTITY file: IDENTITY 2024-05-06T13:26:44.468 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-05-06T13:26:44.468 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-05-06T13:26:44.468 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 83136 ; 2024-05-06T13:26:44.468 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.error_if_exists: 0 2024-05-06T13:26:44.468 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.create_if_missing: 0 2024-05-06T13:26:44.468 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.paranoid_checks: 1 2024-05-06T13:26:44.468 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-06T13:26:44.468 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.env: 0x560d892c6860 2024-05-06T13:26:44.468 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.fs: Posix File System 2024-05-06T13:26:44.468 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.info_log: 0x560d8b0bbdc0 2024-05-06T13:26:44.468 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.max_file_opening_threads: 16 2024-05-06T13:26:44.468 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.statistics: (nil) 2024-05-06T13:26:44.468 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.use_fsync: 0 2024-05-06T13:26:44.468 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.max_log_file_size: 0 2024-05-06T13:26:44.468 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-06T13:26:44.468 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-06T13:26:44.469 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.keep_log_file_num: 1000 2024-05-06T13:26:44.469 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.recycle_log_file_num: 0 2024-05-06T13:26:44.469 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.allow_fallocate: 1 2024-05-06T13:26:44.469 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.allow_mmap_reads: 0 2024-05-06T13:26:44.469 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.allow_mmap_writes: 0 2024-05-06T13:26:44.469 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.use_direct_reads: 0 2024-05-06T13:26:44.469 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-06T13:26:44.469 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.create_missing_column_families: 0 2024-05-06T13:26:44.469 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.db_log_dir: 2024-05-06T13:26:44.469 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-05-06T13:26:44.469 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-06T13:26:44.469 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-06T13:26:44.469 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-06T13:26:44.469 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-06T13:26:44.469 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-06T13:26:44.470 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-06T13:26:44.470 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.advise_random_on_open: 1 2024-05-06T13:26:44.471 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.db_write_buffer_size: 0 2024-05-06T13:26:44.471 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.write_buffer_manager: 0x560d8b1ac2a0 2024-05-06T13:26:44.471 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-06T13:26:44.471 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-05-06T13:26:44.471 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-06T13:26:44.471 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-06T13:26:44.472 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.rate_limiter: (nil) 2024-05-06T13:26:44.472 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-06T13:26:44.472 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.wal_recovery_mode: 2 2024-05-06T13:26:44.472 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.enable_thread_tracking: 0 2024-05-06T13:26:44.472 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.enable_pipelined_write: 0 2024-05-06T13:26:44.472 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.unordered_write: 0 2024-05-06T13:26:44.472 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-06T13:26:44.472 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-06T13:26:44.472 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-06T13:26:44.472 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-06T13:26:44.472 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.row_cache: None 2024-05-06T13:26:44.472 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.wal_filter: None 2024-05-06T13:26:44.472 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-06T13:26:44.472 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.allow_ingest_behind: 0 2024-05-06T13:26:44.472 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.preserve_deletes: 0 2024-05-06T13:26:44.472 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.two_write_queues: 0 2024-05-06T13:26:44.473 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.manual_wal_flush: 0 2024-05-06T13:26:44.473 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.atomic_flush: 0 2024-05-06T13:26:44.473 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-06T13:26:44.473 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-06T13:26:44.473 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-06T13:26:44.473 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.log_readahead_size: 0 2024-05-06T13:26:44.473 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-06T13:26:44.473 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.best_efforts_recovery: 0 2024-05-06T13:26:44.473 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-06T13:26:44.473 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-06T13:26:44.473 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.allow_data_in_errors: 0 2024-05-06T13:26:44.473 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.db_host_id: __hostname__ 2024-05-06T13:26:44.473 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.max_background_jobs: 2 2024-05-06T13:26:44.473 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.max_background_compactions: -1 2024-05-06T13:26:44.473 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.max_subcompactions: 1 2024-05-06T13:26:44.473 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-06T13:26:44.474 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-06T13:26:44.474 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-06T13:26:44.474 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.max_total_wal_size: 0 2024-05-06T13:26:44.474 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-06T13:26:44.474 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-06T13:26:44.474 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-06T13:26:44.474 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-06T13:26:44.474 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.max_open_files: -1 2024-05-06T13:26:44.474 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.bytes_per_sync: 0 2024-05-06T13:26:44.474 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-06T13:26:44.474 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-06T13:26:44.474 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.compaction_readahead_size: 0 2024-05-06T13:26:44.474 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.max_background_flushes: -1 2024-05-06T13:26:44.474 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Compression algorithms supported: 2024-05-06T13:26:44.474 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-06T13:26:44.474 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: kZSTD supported: 0 2024-05-06T13:26:44.475 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: kXpressCompression supported: 0 2024-05-06T13:26:44.475 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: kLZ4HCCompression supported: 1 2024-05-06T13:26:44.475 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: kLZ4Compression supported: 1 2024-05-06T13:26:44.475 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: kBZip2Compression supported: 0 2024-05-06T13:26:44.475 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: kZlibCompression supported: 1 2024-05-06T13:26:44.475 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: kSnappyCompression supported: 1 2024-05-06T13:26:44.475 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-06T13:26:44.475 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-05-06T13:26:44.475 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-05-06T13:26:44.475 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-06T13:26:44.475 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.merge_operator: 2024-05-06T13:26:44.475 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.compaction_filter: None 2024-05-06T13:26:44.476 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.compaction_filter_factory: None 2024-05-06T13:26:44.476 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.sst_partitioner_factory: None 2024-05-06T13:26:44.476 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-06T13:26:44.476 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-06T13:26:44.476 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560d8b087d00) 2024-05-06T13:26:44.476 INFO:journalctl@ceph.mon.a.smithi077.stdout: cache_index_and_filter_blocks: 1 2024-05-06T13:26:44.476 INFO:journalctl@ceph.mon.a.smithi077.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-06T13:26:44.476 INFO:journalctl@ceph.mon.a.smithi077.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-06T13:26:44.476 INFO:journalctl@ceph.mon.a.smithi077.stdout: pin_top_level_index_and_filter: 1 2024-05-06T13:26:44.477 INFO:journalctl@ceph.mon.a.smithi077.stdout: index_type: 0 2024-05-06T13:26:44.477 INFO:journalctl@ceph.mon.a.smithi077.stdout: data_block_index_type: 0 2024-05-06T13:26:44.477 INFO:journalctl@ceph.mon.a.smithi077.stdout: index_shortening: 1 2024-05-06T13:26:44.477 INFO:journalctl@ceph.mon.a.smithi077.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-06T13:26:44.477 INFO:journalctl@ceph.mon.a.smithi077.stdout: hash_index_allow_collision: 1 2024-05-06T13:26:44.477 INFO:journalctl@ceph.mon.a.smithi077.stdout: checksum: 1 2024-05-06T13:26:44.477 INFO:journalctl@ceph.mon.a.smithi077.stdout: no_block_cache: 0 2024-05-06T13:26:44.477 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_cache: 0x560d8b0f2170 2024-05-06T13:26:44.477 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_cache_name: BinnedLRUCache 2024-05-06T13:26:44.477 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_cache_options: 2024-05-06T13:26:44.477 INFO:journalctl@ceph.mon.a.smithi077.stdout: capacity : 536870912 2024-05-06T13:26:44.477 INFO:journalctl@ceph.mon.a.smithi077.stdout: num_shard_bits : 4 2024-05-06T13:26:44.477 INFO:journalctl@ceph.mon.a.smithi077.stdout: strict_capacity_limit : 0 2024-05-06T13:26:44.477 INFO:journalctl@ceph.mon.a.smithi077.stdout: high_pri_pool_ratio: 0.000 2024-05-06T13:26:44.477 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_cache_compressed: (nil) 2024-05-06T13:26:44.478 INFO:journalctl@ceph.mon.a.smithi077.stdout: persistent_cache: (nil) 2024-05-06T13:26:44.478 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_size: 4096 2024-05-06T13:26:44.478 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_size_deviation: 10 2024-05-06T13:26:44.478 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_restart_interval: 16 2024-05-06T13:26:44.478 INFO:journalctl@ceph.mon.a.smithi077.stdout: index_block_restart_interval: 1 2024-05-06T13:26:44.478 INFO:journalctl@ceph.mon.a.smithi077.stdout: metadata_block_size: 4096 2024-05-06T13:26:44.478 INFO:journalctl@ceph.mon.a.smithi077.stdout: partition_filters: 0 2024-05-06T13:26:44.478 INFO:journalctl@ceph.mon.a.smithi077.stdout: use_delta_encoding: 1 2024-05-06T13:26:44.478 INFO:journalctl@ceph.mon.a.smithi077.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-05-06T13:26:44.478 INFO:journalctl@ceph.mon.a.smithi077.stdout: whole_key_filtering: 1 2024-05-06T13:26:44.478 INFO:journalctl@ceph.mon.a.smithi077.stdout: verify_compression: 0 2024-05-06T13:26:44.478 INFO:journalctl@ceph.mon.a.smithi077.stdout: read_amp_bytes_per_bit: 0 2024-05-06T13:26:44.478 INFO:journalctl@ceph.mon.a.smithi077.stdout: format_version: 4 2024-05-06T13:26:44.478 INFO:journalctl@ceph.mon.a.smithi077.stdout: enable_index_compression: 1 2024-05-06T13:26:44.478 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_align: 0 2024-05-06T13:26:44.478 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.write_buffer_size: 33554432 2024-05-06T13:26:44.479 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.max_write_buffer_number: 2 2024-05-06T13:26:44.479 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.compression: NoCompression 2024-05-06T13:26:44.479 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.bottommost_compression: Disabled 2024-05-06T13:26:44.479 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.prefix_extractor: nullptr 2024-05-06T13:26:44.479 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-06T13:26:44.479 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.num_levels: 7 2024-05-06T13:26:44.479 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-06T13:26:44.479 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-06T13:26:44.479 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-06T13:26:44.479 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-06T13:26:44.479 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-06T13:26:44.479 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-06T13:26:44.479 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-06T13:26:44.479 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-06T13:26:44.479 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-06T13:26:44.479 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-06T13:26:44.480 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-06T13:26:44.480 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.compression_opts.level: 32767 2024-05-06T13:26:44.480 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.compression_opts.strategy: 0 2024-05-06T13:26:44.480 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-06T13:26:44.480 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-06T13:26:44.480 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-06T13:26:44.480 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.compression_opts.enabled: false 2024-05-06T13:26:44.480 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-06T13:26:44.480 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-06T13:26:44.480 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-06T13:26:44.480 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.target_file_size_base: 67108864 2024-05-06T13:26:44.480 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-06T13:26:44.480 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-06T13:26:44.481 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-06T13:26:44.481 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-06T13:26:44.481 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-06T13:26:44.481 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-06T13:26:44.481 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-06T13:26:44.481 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-06T13:26:44.481 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-06T13:26:44.481 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-06T13:26:44.481 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-06T13:26:44.481 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-06T13:26:44.481 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-06T13:26:44.481 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.arena_block_size: 4194304 2024-05-06T13:26:44.481 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-06T13:26:44.481 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-06T13:26:44.482 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-05-06T13:26:44.482 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.disable_auto_compactions: 0 2024-05-06T13:26:44.482 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-06T13:26:44.482 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-06T13:26:44.482 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-06T13:26:44.482 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-06T13:26:44.482 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-06T13:26:44.482 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-06T13:26:44.482 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-06T13:26:44.482 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-06T13:26:44.482 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-06T13:26:44.482 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-06T13:26:44.483 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.table_properties_collectors: 2024-05-06T13:26:44.483 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.inplace_update_support: 0 2024-05-06T13:26:44.483 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-06T13:26:44.483 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-06T13:26:44.483 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-06T13:26:44.483 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-06T13:26:44.483 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.bloom_locality: 0 2024-05-06T13:26:44.483 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.max_successive_merges: 0 2024-05-06T13:26:44.483 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-06T13:26:44.483 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.paranoid_file_checks: 0 2024-05-06T13:26:44.483 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.force_consistency_checks: 1 2024-05-06T13:26:44.483 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.report_bg_io_stats: 0 2024-05-06T13:26:44.483 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.ttl: 2592000 2024-05-06T13:26:44.483 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-06T13:26:44.484 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.enable_blob_files: false 2024-05-06T13:26:44.484 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.min_blob_size: 0 2024-05-06T13:26:44.484 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.blob_file_size: 268435456 2024-05-06T13:26:44.484 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-06T13:26:44.484 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-06T13:26:44.484 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-06T13:26:44.484 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-06T13:26:44.484 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-05-06T13:26:44.484 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-05-06T13:26:44.484 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715002004407989, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-05-06T13:26:44.484 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-05-06T13:26:44.484 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-05-06T13:26:44.484 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715002004408676, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79442, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77705, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10043, "raw_average_key_size": 47, "raw_value_size": 72001, "raw_average_value_size": 339, "num_data_blocks": 10, "num_entries": 212, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1715002004, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "fe598134-8211-495d-9b79-a1c20296cd97", "db_session_id": "GE7NOLOHLYPLH2NUSB97"}} 2024-05-06T13:26:44.484 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-05-06T13:26:44.484 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715002004409146, "job": 1, "event": "recovery_finished"} 2024-05-06T13:26:44.485 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-06T13:26:44.485 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x560d8b0d8a80 2024-05-06T13:26:44.485 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: DB pointer 0x560d8b0e8000 2024-05-06T13:26:44.485 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-05-06T13:26:44.485 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-05-06T13:26:44.485 INFO:journalctl@ceph.mon.a.smithi077.stdout: ** DB Stats ** 2024-05-06T13:26:44.485 INFO:journalctl@ceph.mon.a.smithi077.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T13:26:44.485 INFO:journalctl@ceph.mon.a.smithi077.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-06T13:26:44.485 INFO:journalctl@ceph.mon.a.smithi077.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-06T13:26:44.485 INFO:journalctl@ceph.mon.a.smithi077.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-06T13:26:44.485 INFO:journalctl@ceph.mon.a.smithi077.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-06T13:26:44.485 INFO:journalctl@ceph.mon.a.smithi077.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-05-06T13:26:44.485 INFO:journalctl@ceph.mon.a.smithi077.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-06T13:26:44.485 INFO:journalctl@ceph.mon.a.smithi077.stdout: 2024-05-06T13:26:44.485 INFO:journalctl@ceph.mon.a.smithi077.stdout: ** Compaction Stats [default] ** 2024-05-06T13:26:44.485 INFO:journalctl@ceph.mon.a.smithi077.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-06T13:26:44.486 INFO:journalctl@ceph.mon.a.smithi077.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-06T13:26:44.486 INFO:journalctl@ceph.mon.a.smithi077.stdout: L0 2/0 79.34 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 166.1 0.00 0.00 1 0.000 0 0 2024-05-06T13:26:44.486 INFO:journalctl@ceph.mon.a.smithi077.stdout: Sum 2/0 79.34 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 166.1 0.00 0.00 1 0.000 0 0 2024-05-06T13:26:44.486 INFO:journalctl@ceph.mon.a.smithi077.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 166.1 0.00 0.00 1 0.000 0 0 2024-05-06T13:26:44.487 INFO:journalctl@ceph.mon.a.smithi077.stdout: 2024-05-06T13:26:44.487 INFO:journalctl@ceph.mon.a.smithi077.stdout: ** Compaction Stats [default] ** 2024-05-06T13:26:44.487 INFO:journalctl@ceph.mon.a.smithi077.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-06T13:26:44.487 INFO:journalctl@ceph.mon.a.smithi077.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-06T13:26:44.487 INFO:journalctl@ceph.mon.a.smithi077.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 166.1 0.00 0.00 1 0.000 0 0 2024-05-06T13:26:44.487 INFO:journalctl@ceph.mon.a.smithi077.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T13:26:44.487 INFO:journalctl@ceph.mon.a.smithi077.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-06T13:26:44.487 INFO:journalctl@ceph.mon.a.smithi077.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-06T13:26:44.487 INFO:journalctl@ceph.mon.a.smithi077.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-06T13:26:44.487 INFO:journalctl@ceph.mon.a.smithi077.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-06T13:26:44.487 INFO:journalctl@ceph.mon.a.smithi077.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-06T13:26:44.487 INFO:journalctl@ceph.mon.a.smithi077.stdout: Cumulative compaction: 0.00 GB write, 25.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T13:26:44.487 INFO:journalctl@ceph.mon.a.smithi077.stdout: Interval compaction: 0.00 GB write, 25.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T13:26:44.487 INFO:journalctl@ceph.mon.a.smithi077.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-06T13:26:44.488 INFO:journalctl@ceph.mon.a.smithi077.stdout: 2024-05-06T13:26:44.488 INFO:journalctl@ceph.mon.a.smithi077.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-06T13:26:44.488 INFO:journalctl@ceph.mon.a.smithi077.stdout: 2024-05-06T13:26:44.488 INFO:journalctl@ceph.mon.a.smithi077.stdout: ** Compaction Stats [default] ** 2024-05-06T13:26:44.488 INFO:journalctl@ceph.mon.a.smithi077.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-06T13:26:44.488 INFO:journalctl@ceph.mon.a.smithi077.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-06T13:26:44.488 INFO:journalctl@ceph.mon.a.smithi077.stdout: L0 2/0 79.34 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 166.1 0.00 0.00 1 0.000 0 0 2024-05-06T13:26:44.488 INFO:journalctl@ceph.mon.a.smithi077.stdout: Sum 2/0 79.34 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 166.1 0.00 0.00 1 0.000 0 0 2024-05-06T13:26:44.488 INFO:journalctl@ceph.mon.a.smithi077.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-05-06T13:26:44.488 INFO:journalctl@ceph.mon.a.smithi077.stdout: 2024-05-06T13:26:44.488 INFO:journalctl@ceph.mon.a.smithi077.stdout: ** Compaction Stats [default] ** 2024-05-06T13:26:44.488 INFO:journalctl@ceph.mon.a.smithi077.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-06T13:26:44.489 INFO:journalctl@ceph.mon.a.smithi077.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-06T13:26:44.489 INFO:journalctl@ceph.mon.a.smithi077.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 166.1 0.00 0.00 1 0.000 0 0 2024-05-06T13:26:44.489 INFO:journalctl@ceph.mon.a.smithi077.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T13:26:44.489 INFO:journalctl@ceph.mon.a.smithi077.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-06T13:26:44.489 INFO:journalctl@ceph.mon.a.smithi077.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-06T13:26:44.489 INFO:journalctl@ceph.mon.a.smithi077.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-06T13:26:44.489 INFO:journalctl@ceph.mon.a.smithi077.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-06T13:26:44.489 INFO:journalctl@ceph.mon.a.smithi077.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-06T13:26:44.489 INFO:journalctl@ceph.mon.a.smithi077.stdout: Cumulative compaction: 0.00 GB write, 25.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T13:26:44.489 INFO:journalctl@ceph.mon.a.smithi077.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T13:26:44.489 INFO:journalctl@ceph.mon.a.smithi077.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-06T13:26:44.489 INFO:journalctl@ceph.mon.a.smithi077.stdout: 2024-05-06T13:26:44.489 INFO:journalctl@ceph.mon.a.smithi077.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-06T13:26:44.489 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: starting mon.a rank 0 at public addrs [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] at bind addrs [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:26:44.489 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: mon.a@-1(???) e1 preinit fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:26:44.490 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: mon.a@-1(???).mds e1 new map 2024-05-06T13:26:44.490 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: mon.a@-1(???).mds e1 print_map 2024-05-06T13:26:44.490 INFO:journalctl@ceph.mon.a.smithi077.stdout: e1 2024-05-06T13:26:44.490 INFO:journalctl@ceph.mon.a.smithi077.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-06T13:26:44.490 INFO:journalctl@ceph.mon.a.smithi077.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-06T13:26:44.490 INFO:journalctl@ceph.mon.a.smithi077.stdout: legacy client fscid: -1 2024-05-06T13:26:44.490 INFO:journalctl@ceph.mon.a.smithi077.stdout: 2024-05-06T13:26:44.490 INFO:journalctl@ceph.mon.a.smithi077.stdout: No filesystems configured 2024-05-06T13:26:44.490 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-06T13:26:44.490 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-06T13:26:44.490 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-06T13:26:44.490 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-06T13:26:44.490 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-05-06T13:26:44.490 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: expand_channel_meta expand map: {default=false} 2024-05-06T13:26:44.490 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: expand_channel_meta from 'false' to 'false' 2024-05-06T13:26:44.491 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: expand_channel_meta expanded map: {default=false} 2024-05-06T13:26:44.491 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: expand_channel_meta expand map: {default=info} 2024-05-06T13:26:44.491 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: expand_channel_meta from 'info' to 'info' 2024-05-06T13:26:44.491 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: expand_channel_meta expanded map: {default=info} 2024-05-06T13:26:44.491 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: expand_channel_meta expand map: {default=daemon} 2024-05-06T13:26:44.491 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: expand_channel_meta from 'daemon' to 'daemon' 2024-05-06T13:26:44.491 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: expand_channel_meta expanded map: {default=daemon} 2024-05-06T13:26:44.491 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: expand_channel_meta expand map: {default=debug} 2024-05-06T13:26:44.491 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: expand_channel_meta from 'debug' to 'debug' 2024-05-06T13:26:44.491 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: expand_channel_meta expanded map: {default=debug} 2024-05-06T13:26:44.491 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 bash[22409]: ac46f91694f871a4c86d9233330db0effb6a0b7cdd415dbb8c13ed18d2c6e0a7 2024-05-06T13:26:44.491 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 systemd[1]: Started Ceph mon.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:26:44.491 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-06T13:26:44.491 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: monmap e1: 1 mons at {a=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0]} 2024-05-06T13:26:44.491 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: fsmap 2024-05-06T13:26:44.491 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: osdmap e1: 0 total, 0 up, 0 in 2024-05-06T13:26:44.492 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:44 smithi077 ceph-mon[22428]: mgrmap e1: no daemons active 2024-05-06T13:26:45.513 INFO:teuthology.orchestra.run.smithi077.stderr:Wrote config to /etc/ceph/ceph.conf 2024-05-06T13:26:45.514 INFO:teuthology.orchestra.run.smithi077.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-05-06T13:26:45.514 INFO:teuthology.orchestra.run.smithi077.stderr:Creating mgr... 2024-05-06T13:26:45.514 INFO:teuthology.orchestra.run.smithi077.stderr:Verifying port 9283 ... 2024-05-06T13:26:45.686 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Failed to reset failed state of unit ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mgr.y.service: Unit ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mgr.y.service not loaded. 2024-05-06T13:26:45.693 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Created symlink /etc/systemd/system/ceph-20e59cb4-0bac-11ef-bc97-c7b262605968.target.wants/ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@.service. 2024-05-06T13:26:46.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:45 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3776430626' entity='client.admin' 2024-05-06T13:26:46.364 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:46 smithi077 podman[22645]: 2024-05-06 13:26:46.267500445 +0000 UTC m=+0.311763479 container start 79a55d163d052c84d979db49ef7f91554cf70486bb677633de3378378d731f8d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, version=8, vcs-type=git, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-05-06T13:26:46.364 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:46 smithi077 bash[22645]: 79a55d163d052c84d979db49ef7f91554cf70486bb677633de3378378d731f8d 2024-05-06T13:26:46.377 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: disabled 2024-05-06T13:26:46.384 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: inactive 2024-05-06T13:26:46.385 INFO:teuthology.orchestra.run.smithi077.stderr:firewalld.service is not enabled 2024-05-06T13:26:46.385 INFO:teuthology.orchestra.run.smithi077.stderr:Not possible to enable service . firewalld.service is not available 2024-05-06T13:26:46.393 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: disabled 2024-05-06T13:26:46.399 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: inactive 2024-05-06T13:26:46.400 INFO:teuthology.orchestra.run.smithi077.stderr:firewalld.service is not enabled 2024-05-06T13:26:46.400 INFO:teuthology.orchestra.run.smithi077.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-05-06T13:26:46.401 INFO:teuthology.orchestra.run.smithi077.stderr:Waiting for mgr to start... 2024-05-06T13:26:46.401 INFO:teuthology.orchestra.run.smithi077.stderr:Waiting for mgr... 2024-05-06T13:26:46.639 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:46 smithi077 systemd[1]: Started Ceph mgr.y for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:26:46.640 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:46 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:46.410+0000 7fcd47190000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T13:26:46.640 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:46 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:46.542+0000 7fcd47190000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:26:46.640 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:46 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:46.641+0000 7fcd47190000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T13:26:47.082 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-05-06T13:26:47.082 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: { 2024-05-06T13:26:47.082 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "fsid": "20e59cb4-0bac-11ef-bc97-c7b262605968", 2024-05-06T13:26:47.082 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "health": { 2024-05-06T13:26:47.082 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-05-06T13:26:47.082 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "checks": {}, 2024-05-06T13:26:47.082 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "mutes": [] 2024-05-06T13:26:47.082 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-05-06T13:26:47.082 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-05-06T13:26:47.082 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "quorum": [ 2024-05-06T13:26:47.082 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 0 2024-05-06T13:26:47.083 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ], 2024-05-06T13:26:47.083 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "quorum_names": [ 2024-05-06T13:26:47.083 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "a" 2024-05-06T13:26:47.083 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ], 2024-05-06T13:26:47.083 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-05-06T13:26:47.083 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "monmap": { 2024-05-06T13:26:47.083 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-06T13:26:47.083 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-05-06T13:26:47.083 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_mons": 1 2024-05-06T13:26:47.083 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-05-06T13:26:47.083 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "osdmap": { 2024-05-06T13:26:47.083 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-06T13:26:47.083 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_osds": 0, 2024-05-06T13:26:47.083 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-05-06T13:26:47.083 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-05-06T13:26:47.083 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-05-06T13:26:47.087 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-05-06T13:26:47.087 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-05-06T13:26:47.087 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-05-06T13:26:47.087 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "pgmap": { 2024-05-06T13:26:47.087 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-05-06T13:26:47.087 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-05-06T13:26:47.087 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_pools": 0, 2024-05-06T13:26:47.087 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_objects": 0, 2024-05-06T13:26:47.087 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-05-06T13:26:47.087 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-05-06T13:26:47.088 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-05-06T13:26:47.088 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "bytes_total": 0 2024-05-06T13:26:47.088 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-05-06T13:26:47.088 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "fsmap": { 2024-05-06T13:26:47.088 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-06T13:26:47.088 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "by_rank": [], 2024-05-06T13:26:47.088 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "up:standby": 0 2024-05-06T13:26:47.088 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-05-06T13:26:47.088 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "mgrmap": { 2024-05-06T13:26:47.088 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "available": false, 2024-05-06T13:26:47.088 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-05-06T13:26:47.088 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "modules": [ 2024-05-06T13:26:47.088 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "iostat", 2024-05-06T13:26:47.088 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "nfs", 2024-05-06T13:26:47.088 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "restful" 2024-05-06T13:26:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ], 2024-05-06T13:26:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "services": {} 2024-05-06T13:26:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-05-06T13:26:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "servicemap": { 2024-05-06T13:26:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-06T13:26:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "modified": "2024-05-06T13:26:37.391628+0000", 2024-05-06T13:26:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "services": {} 2024-05-06T13:26:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-05-06T13:26:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "progress_events": {} 2024-05-06T13:26:47.089 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: } 2024-05-06T13:26:47.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:47 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/889258358' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-06T13:26:47.301 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:46 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:46.954+0000 7fcd47190000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T13:26:47.301 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:47 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:47.039+0000 7fcd47190000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T13:26:47.664 INFO:teuthology.orchestra.run.smithi077.stderr:mgr not available, waiting (1/15)... 2024-05-06T13:26:48.551 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:48 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:48.179+0000 7fcd47190000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T13:26:48.551 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:48 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:48.343+0000 7fcd47190000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T13:26:48.551 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:48 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:48.424+0000 7fcd47190000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T13:26:49.051 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:48 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:48.732+0000 7fcd47190000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T13:26:49.051 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:48 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:48.821+0000 7fcd47190000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T13:26:49.051 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:48 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:48.912+0000 7fcd47190000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T13:26:49.551 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:49 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:49.148+0000 7fcd47190000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:26:49.894 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:49 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:49.579+0000 7fcd47190000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T13:26:50.165 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:50 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:50.070+0000 7fcd47190000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T13:26:50.194 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-05-06T13:26:50.194 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: { 2024-05-06T13:26:50.194 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "fsid": "20e59cb4-0bac-11ef-bc97-c7b262605968", 2024-05-06T13:26:50.194 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "health": { 2024-05-06T13:26:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-05-06T13:26:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "checks": {}, 2024-05-06T13:26:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "mutes": [] 2024-05-06T13:26:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-05-06T13:26:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-05-06T13:26:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "quorum": [ 2024-05-06T13:26:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 0 2024-05-06T13:26:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ], 2024-05-06T13:26:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "quorum_names": [ 2024-05-06T13:26:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "a" 2024-05-06T13:26:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ], 2024-05-06T13:26:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-05-06T13:26:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "monmap": { 2024-05-06T13:26:50.195 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-06T13:26:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-05-06T13:26:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_mons": 1 2024-05-06T13:26:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-05-06T13:26:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "osdmap": { 2024-05-06T13:26:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-06T13:26:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_osds": 0, 2024-05-06T13:26:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-05-06T13:26:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-05-06T13:26:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-05-06T13:26:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-05-06T13:26:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-05-06T13:26:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-05-06T13:26:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "pgmap": { 2024-05-06T13:26:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-05-06T13:26:50.196 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-05-06T13:26:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_pools": 0, 2024-05-06T13:26:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_objects": 0, 2024-05-06T13:26:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-05-06T13:26:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-05-06T13:26:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-05-06T13:26:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "bytes_total": 0 2024-05-06T13:26:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-05-06T13:26:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "fsmap": { 2024-05-06T13:26:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-06T13:26:50.197 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "by_rank": [], 2024-05-06T13:26:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "up:standby": 0 2024-05-06T13:26:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-05-06T13:26:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "mgrmap": { 2024-05-06T13:26:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "available": false, 2024-05-06T13:26:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-05-06T13:26:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "modules": [ 2024-05-06T13:26:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "iostat", 2024-05-06T13:26:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "nfs", 2024-05-06T13:26:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "restful" 2024-05-06T13:26:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ], 2024-05-06T13:26:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "services": {} 2024-05-06T13:26:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-05-06T13:26:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "servicemap": { 2024-05-06T13:26:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-06T13:26:50.198 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "modified": "2024-05-06T13:26:37.391628+0000", 2024-05-06T13:26:50.199 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "services": {} 2024-05-06T13:26:50.199 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-05-06T13:26:50.199 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "progress_events": {} 2024-05-06T13:26:50.199 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: } 2024-05-06T13:26:50.465 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:50 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/1857539211' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-06T13:26:50.466 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:50 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:50.167+0000 7fcd47190000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T13:26:50.466 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:50 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:50.467+0000 7fcd47190000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T13:26:50.791 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:50 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:50.556+0000 7fcd47190000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T13:26:50.864 INFO:teuthology.orchestra.run.smithi077.stderr:mgr not available, waiting (2/15)... 2024-05-06T13:26:51.051 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:50 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:50.956+0000 7fcd47190000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T13:26:51.377 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:51 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:51.087+0000 7fcd47190000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T13:26:51.377 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:51 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:51.179+0000 7fcd47190000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T13:26:51.377 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:51 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:51.290+0000 7fcd47190000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T13:26:51.377 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:51 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:51.378+0000 7fcd47190000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T13:26:52.269 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:51 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:51.916+0000 7fcd47190000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T13:26:52.551 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:52 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:52.270+0000 7fcd47190000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T13:26:52.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:52 smithi077 ceph-mon[22428]: Activating manager daemon y 2024-05-06T13:26:52.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:52 smithi077 ceph-mon[22428]: mgrmap e2: y(active, starting, since 0.00303351s) 2024-05-06T13:26:52.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:52 smithi077 ceph-mon[22428]: from='mgr.14100 172.21.15.77:0/4000433697' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T13:26:52.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:52 smithi077 ceph-mon[22428]: from='mgr.14100 172.21.15.77:0/4000433697' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T13:26:52.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:52 smithi077 ceph-mon[22428]: from='mgr.14100 172.21.15.77:0/4000433697' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T13:26:52.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:52 smithi077 ceph-mon[22428]: from='mgr.14100 172.21.15.77:0/4000433697' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:26:52.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:52 smithi077 ceph-mon[22428]: from='mgr.14100 172.21.15.77:0/4000433697' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T13:26:52.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:52 smithi077 ceph-mon[22428]: Manager daemon y is now available 2024-05-06T13:26:52.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:52 smithi077 ceph-mon[22428]: from='mgr.14100 172.21.15.77:0/4000433697' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:26:52.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:52 smithi077 ceph-mon[22428]: from='mgr.14100 172.21.15.77:0/4000433697' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:26:52.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:52 smithi077 ceph-mon[22428]: from='mgr.14100 172.21.15.77:0/4000433697' entity='mgr.y' 2024-05-06T13:26:52.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:52 smithi077 ceph-mon[22428]: from='mgr.14100 172.21.15.77:0/4000433697' entity='mgr.y' 2024-05-06T13:26:52.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:52 smithi077 ceph-mon[22428]: from='mgr.14100 172.21.15.77:0/4000433697' entity='mgr.y' 2024-05-06T13:26:53.541 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-05-06T13:26:53.541 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: { 2024-05-06T13:26:53.541 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "fsid": "20e59cb4-0bac-11ef-bc97-c7b262605968", 2024-05-06T13:26:53.542 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "health": { 2024-05-06T13:26:53.542 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-05-06T13:26:53.542 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "checks": {}, 2024-05-06T13:26:53.542 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "mutes": [] 2024-05-06T13:26:53.542 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-05-06T13:26:53.542 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-05-06T13:26:53.542 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "quorum": [ 2024-05-06T13:26:53.542 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 0 2024-05-06T13:26:53.542 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ], 2024-05-06T13:26:53.542 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "quorum_names": [ 2024-05-06T13:26:53.542 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "a" 2024-05-06T13:26:53.542 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ], 2024-05-06T13:26:53.542 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "quorum_age": 9, 2024-05-06T13:26:53.542 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "monmap": { 2024-05-06T13:26:53.542 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-06T13:26:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-05-06T13:26:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_mons": 1 2024-05-06T13:26:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-05-06T13:26:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "osdmap": { 2024-05-06T13:26:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-06T13:26:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_osds": 0, 2024-05-06T13:26:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-05-06T13:26:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-05-06T13:26:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-05-06T13:26:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-05-06T13:26:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-05-06T13:26:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-05-06T13:26:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "pgmap": { 2024-05-06T13:26:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-05-06T13:26:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-05-06T13:26:53.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_pools": 0, 2024-05-06T13:26:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_objects": 0, 2024-05-06T13:26:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-05-06T13:26:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-05-06T13:26:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-05-06T13:26:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "bytes_total": 0 2024-05-06T13:26:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-05-06T13:26:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "fsmap": { 2024-05-06T13:26:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-06T13:26:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "by_rank": [], 2024-05-06T13:26:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "up:standby": 0 2024-05-06T13:26:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-05-06T13:26:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "mgrmap": { 2024-05-06T13:26:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "available": true, 2024-05-06T13:26:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-05-06T13:26:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "modules": [ 2024-05-06T13:26:53.544 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "iostat", 2024-05-06T13:26:53.545 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "nfs", 2024-05-06T13:26:53.545 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "restful" 2024-05-06T13:26:53.545 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ], 2024-05-06T13:26:53.545 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "services": {} 2024-05-06T13:26:53.545 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-05-06T13:26:53.545 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "servicemap": { 2024-05-06T13:26:53.545 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-05-06T13:26:53.546 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "modified": "2024-05-06T13:26:37.391628+0000", 2024-05-06T13:26:53.546 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "services": {} 2024-05-06T13:26:53.546 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-05-06T13:26:53.546 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "progress_events": {} 2024-05-06T13:26:53.546 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: } 2024-05-06T13:26:54.103 INFO:teuthology.orchestra.run.smithi077.stderr:mgr is available 2024-05-06T13:26:54.529 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:54 smithi077 ceph-mon[22428]: mgrmap e3: y(active, since 1.00766s) 2024-05-06T13:26:54.529 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:54 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/4081279330' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-06T13:26:54.740 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-05-06T13:26:54.741 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: [global] 2024-05-06T13:26:54.741 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: fsid = 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:26:54.741 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-05-06T13:26:54.741 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-05-06T13:26:54.741 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-05-06T13:26:54.741 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-05-06T13:26:54.741 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-05-06T13:26:54.741 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: [mgr] 2024-05-06T13:26:54.741 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-05-06T13:26:54.741 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-05-06T13:26:54.741 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: [osd] 2024-05-06T13:26:54.741 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-05-06T13:26:54.742 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-05-06T13:26:55.276 INFO:teuthology.orchestra.run.smithi077.stderr:Enabling cephadm module... 2024-05-06T13:26:55.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:55 smithi077 ceph-mon[22428]: mgrmap e4: y(active, since 2s) 2024-05-06T13:26:55.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:55 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/483883626' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-06T13:26:55.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:55 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/483883626' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-06T13:26:56.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:56 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3264000791' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-06T13:26:56.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:56 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ignoring --setuser ceph since I am not root 2024-05-06T13:26:56.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:56 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ignoring --setgroup ceph since I am not root 2024-05-06T13:26:56.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:56 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:56.444+0000 7f4656140000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T13:26:56.872 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:56 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:56.572+0000 7f4656140000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:26:56.872 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:56 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:56.668+0000 7f4656140000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T13:26:57.191 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:56 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:56.974+0000 7f4656140000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T13:26:57.191 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:57 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:57.057+0000 7f4656140000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T13:26:57.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:57 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3264000791' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-06T13:26:57.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:57 smithi077 ceph-mon[22428]: mgrmap e5: y(active, since 4s) 2024-05-06T13:26:57.549 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: { 2024-05-06T13:26:57.550 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 5, 2024-05-06T13:26:57.550 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "available": true, 2024-05-06T13:26:57.550 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "active_name": "y", 2024-05-06T13:26:57.550 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_standby": 0 2024-05-06T13:26:57.550 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: } 2024-05-06T13:26:58.180 INFO:teuthology.orchestra.run.smithi077.stderr:Waiting for the mgr to restart... 2024-05-06T13:26:58.180 INFO:teuthology.orchestra.run.smithi077.stderr:Waiting for mgr epoch 5... 2024-05-06T13:26:58.429 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:26:58 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/4146590142' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-06T13:26:58.429 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:58 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:58.172+0000 7f4656140000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T13:26:58.429 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:58 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:58.345+0000 7f4656140000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T13:26:58.430 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:58 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:58.429+0000 7f4656140000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T13:26:59.051 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:58 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:58.743+0000 7f4656140000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T13:26:59.051 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:58 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:58.830+0000 7f4656140000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T13:26:59.051 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:58 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:58.920+0000 7f4656140000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T13:26:59.551 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:59 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:59.153+0000 7f4656140000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:27:00.047 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:26:59 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:26:59.578+0000 7f4656140000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T13:27:00.047 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:00 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:00.047+0000 7f4656140000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T13:27:00.301 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:00 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:00.138+0000 7f4656140000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T13:27:00.801 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:00 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:00.423+0000 7f4656140000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T13:27:00.802 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:00 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:00.509+0000 7f4656140000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T13:27:01.224 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:00 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:00.898+0000 7f4656140000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T13:27:01.224 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:01 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:01.025+0000 7f4656140000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T13:27:01.224 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:01 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:01.114+0000 7f4656140000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T13:27:01.224 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:01 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:01.224+0000 7f4656140000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T13:27:01.551 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:01 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:01.310+0000 7f4656140000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T13:27:02.196 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:01 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:01.845+0000 7f4656140000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T13:27:02.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:02 smithi077 ceph-mon[22428]: Active manager daemon y restarted 2024-05-06T13:27:02.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:02 smithi077 ceph-mon[22428]: Activating manager daemon y 2024-05-06T13:27:02.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:02 smithi077 ceph-mon[22428]: osdmap e2: 0 total, 0 up, 0 in 2024-05-06T13:27:02.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:02 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:02.197+0000 7f4656140000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T13:27:03.262 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: { 2024-05-06T13:27:03.262 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-05-06T13:27:03.262 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "initialized": true 2024-05-06T13:27:03.262 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: } 2024-05-06T13:27:03.485 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: [06/May/2024:13:27:03] ENGINE Bus STARTING 2024-05-06T13:27:03.485 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: [06/May/2024:13:27:03] ENGINE Serving on https://172.21.15.77:7150 2024-05-06T13:27:03.485 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: [06/May/2024:13:27:03] ENGINE Bus STARTED 2024-05-06T13:27:03.485 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:03 smithi077 ceph-mon[22428]: mgrmap e6: y(active, starting, since 0.0524327s) 2024-05-06T13:27:03.486 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:03 smithi077 ceph-mon[22428]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:27:03.486 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:03 smithi077 ceph-mon[22428]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T13:27:03.486 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:03 smithi077 ceph-mon[22428]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T13:27:03.486 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:03 smithi077 ceph-mon[22428]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T13:27:03.486 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:03 smithi077 ceph-mon[22428]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T13:27:03.486 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:03 smithi077 ceph-mon[22428]: Manager daemon y is now available 2024-05-06T13:27:03.486 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:03 smithi077 ceph-mon[22428]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' 2024-05-06T13:27:03.486 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:03 smithi077 ceph-mon[22428]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' 2024-05-06T13:27:03.486 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:03 smithi077 ceph-mon[22428]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:27:03.486 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:03 smithi077 ceph-mon[22428]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:27:03.486 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:03 smithi077 ceph-mon[22428]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:27:03.486 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:03 smithi077 ceph-mon[22428]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:27:03.486 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:03 smithi077 ceph-mon[22428]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:27:03.817 INFO:teuthology.orchestra.run.smithi077.stderr:mgr epoch 5 is available 2024-05-06T13:27:03.817 INFO:teuthology.orchestra.run.smithi077.stderr:Setting orchestrator backend to cephadm... 2024-05-06T13:27:04.443 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:04 smithi077 ceph-mon[22428]: [06/May/2024:13:27:03] ENGINE Bus STARTING 2024-05-06T13:27:04.444 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:04 smithi077 ceph-mon[22428]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-06T13:27:04.444 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:04 smithi077 ceph-mon[22428]: mgrmap e7: y(active, since 1.05747s) 2024-05-06T13:27:04.444 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:04 smithi077 ceph-mon[22428]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-06T13:27:04.444 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:04 smithi077 ceph-mon[22428]: [06/May/2024:13:27:03] ENGINE Serving on https://172.21.15.77:7150 2024-05-06T13:27:04.444 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:04 smithi077 ceph-mon[22428]: [06/May/2024:13:27:03] ENGINE Bus STARTED 2024-05-06T13:27:04.444 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:04 smithi077 ceph-mon[22428]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' 2024-05-06T13:27:04.444 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:04 smithi077 ceph-mon[22428]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:27:05.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:05 smithi077 ceph-mon[22428]: mgrmap e8: y(active, since 2s) 2024-05-06T13:27:05.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:05 smithi077 ceph-mon[22428]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' 2024-05-06T13:27:05.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:05 smithi077 ceph-mon[22428]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:27:05.573 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: value unchanged 2024-05-06T13:27:06.132 INFO:teuthology.orchestra.run.smithi077.stderr:Generating ssh key... 2024-05-06T13:27:06.380 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:06 smithi077 ceph-mon[22428]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:27:07.419 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:07 smithi077 ceph-mon[22428]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:27:07.419 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:07 smithi077 ceph-mon[22428]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' 2024-05-06T13:27:07.419 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:07 smithi077 ceph-mon[22428]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' 2024-05-06T13:27:07.419 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:07 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: Generating public/private rsa key pair. 2024-05-06T13:27:07.419 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:07 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: Your identification has been saved in /tmp/tmp5uxly5fj/key. 2024-05-06T13:27:07.419 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:07 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: Your public key has been saved in /tmp/tmp5uxly5fj/key.pub. 2024-05-06T13:27:07.419 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:07 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: The key fingerprint is: 2024-05-06T13:27:07.419 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:07 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: SHA256:OK9rhEN/aTMaz4FtsvJ1SSax4I80TZqVSF8LLBDGiSY ceph-20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:27:07.419 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:07 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: The key's randomart image is: 2024-05-06T13:27:07.419 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:07 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: +---[RSA 3072]----+ 2024-05-06T13:27:07.420 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:07 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: | o++... . | 2024-05-06T13:27:07.420 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:07 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: |E o.o..o.+ . | 2024-05-06T13:27:07.420 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:07 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: | o o.* . | 2024-05-06T13:27:07.420 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:07 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: | .. O o | 2024-05-06T13:27:07.420 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:07 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: | . oOoS.o | 2024-05-06T13:27:07.420 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:07 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: | o.**X+ . | 2024-05-06T13:27:07.420 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:07 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: | o.@++o | 2024-05-06T13:27:07.420 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:07 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: | . +oo. | 2024-05-06T13:27:07.420 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:07 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: | ++. | 2024-05-06T13:27:07.420 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:07 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: +----[SHA256]-----+ 2024-05-06T13:27:08.303 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDorU5zm7WNWvVyPWZQgRYoSPB64avdXM6r90t+56c2qjP+kthcLjmo/lKvwg9BP7ywG2H9zT9hpHoDJDnFhXAiy5tDbVZVBCZuTy66YCHmCA814XKimhPfDmiOT+iI2FTWiJNJUr7nzQ62pAxOl56k7q+eI+JD77+sF3kzr1w3qGvJoAof14VvGeE/y8p62Wj20/pWN7EOXr0S9EhDEG0E4y9pllJ3VaXaZ/Qp0xcBDXFoi0uCclh5LLnAertyUntzvzVW3dNO3D8oKluI1eHtWD/5uPqS1/sZQWEFGvUm6ZgU9UboK4hLmKACKN87RPaVgmlyeJQPHovZnfuhwi/sZHTN2+1zuCWznIS9Fj1E0gZRHhaVifaUM5a/4zP3U2VlXF7Jk3C7/Po7Dq9Gs0pp3XxDUT5F7QGnixehyAS1h3c9ac4+7dTWAlB2ZVY3spPkGbqV3DV/dVbLz3/Y6S/w/o21ZCnbGukDE7iwd+N936izTjOP6JKaJdhLNo3bztM= ceph-20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:27:08.801 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:08 smithi077 ceph-mon[22428]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:27:08.801 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:08 smithi077 ceph-mon[22428]: Generating ssh key... 2024-05-06T13:27:08.845 INFO:teuthology.orchestra.run.smithi077.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-05-06T13:27:08.845 INFO:teuthology.orchestra.run.smithi077.stderr:Adding key to root@localhost authorized_keys... 2024-05-06T13:27:08.845 INFO:teuthology.orchestra.run.smithi077.stderr:Adding host smithi077... 2024-05-06T13:27:09.442 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:09 smithi077 ceph-mon[22428]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:27:10.255 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: Added host 'smithi077' with addr '172.21.15.77' 2024-05-06T13:27:10.810 INFO:teuthology.orchestra.run.smithi077.stderr:Deploying unmanaged mon service... 2024-05-06T13:27:11.424 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: Scheduled mon update... 2024-05-06T13:27:11.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:11 smithi077 ceph-mon[22428]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi077", "addr": "172.21.15.77", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:27:11.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:11 smithi077 ceph-mon[22428]: Deploying cephadm binary to smithi077 2024-05-06T13:27:11.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:11 smithi077 ceph-mon[22428]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' 2024-05-06T13:27:11.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:11 smithi077 ceph-mon[22428]: Added host smithi077 2024-05-06T13:27:11.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:11 smithi077 ceph-mon[22428]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:27:12.737 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:12 smithi077 ceph-mon[22428]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:27:12.737 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:12 smithi077 ceph-mon[22428]: Saving service mon spec with placement count:5 2024-05-06T13:27:12.737 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:12 smithi077 ceph-mon[22428]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' 2024-05-06T13:27:12.739 INFO:teuthology.orchestra.run.smithi077.stderr:Deploying unmanaged mgr service... 2024-05-06T13:27:13.324 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-05-06T13:27:14.479 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:14 smithi077 ceph-mon[22428]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:27:14.479 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:14 smithi077 ceph-mon[22428]: Saving service mgr spec with placement count:2 2024-05-06T13:27:14.480 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:14 smithi077 ceph-mon[22428]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' 2024-05-06T13:27:14.480 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:14 smithi077 ceph-mon[22428]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' 2024-05-06T13:27:14.480 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:14 smithi077 ceph-mon[22428]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' 2024-05-06T13:27:16.163 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:15 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2721465418' entity='client.admin' 2024-05-06T13:27:16.903 INFO:teuthology.orchestra.run.smithi077.stderr:Enabling the dashboard module... 2024-05-06T13:27:17.466 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:17 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3709216103' entity='client.admin' 2024-05-06T13:27:17.466 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:17 smithi077 ceph-mon[22428]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' 2024-05-06T13:27:18.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:18 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3574573116' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-06T13:27:18.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:18 smithi077 ceph-mon[22428]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' 2024-05-06T13:27:18.301 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:18 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ignoring --setuser ceph since I am not root 2024-05-06T13:27:18.301 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:18 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ignoring --setgroup ceph since I am not root 2024-05-06T13:27:18.801 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:18 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:18.395+0000 7ff115409000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T13:27:18.801 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:18 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:18.522+0000 7ff115409000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:27:18.801 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:18 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:18.621+0000 7ff115409000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T13:27:19.197 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:18 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:18.928+0000 7ff115409000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T13:27:19.198 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:19 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:19.009+0000 7ff115409000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T13:27:19.457 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:19 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3574573116' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-06T13:27:19.457 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:19 smithi077 ceph-mon[22428]: mgrmap e9: y(active, since 16s) 2024-05-06T13:27:20.230 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: { 2024-05-06T13:27:20.231 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 9, 2024-05-06T13:27:20.231 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "available": true, 2024-05-06T13:27:20.231 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "active_name": "y", 2024-05-06T13:27:20.231 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_standby": 0 2024-05-06T13:27:20.231 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: } 2024-05-06T13:27:20.392 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:20 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:20.142+0000 7ff115409000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T13:27:20.392 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:20 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:20.311+0000 7ff115409000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T13:27:20.392 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:20 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:20.392+0000 7ff115409000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T13:27:20.711 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:20 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2509053377' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-06T13:27:20.827 INFO:teuthology.orchestra.run.smithi077.stderr:Waiting for the mgr to restart... 2024-05-06T13:27:20.827 INFO:teuthology.orchestra.run.smithi077.stderr:Waiting for mgr epoch 9... 2024-05-06T13:27:20.984 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:20 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:20.711+0000 7ff115409000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T13:27:20.984 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:20 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:20.803+0000 7ff115409000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T13:27:20.984 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:20 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:20.900+0000 7ff115409000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T13:27:21.277 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:21 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:21.145+0000 7ff115409000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:27:22.051 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:21 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:21.584+0000 7ff115409000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T13:27:22.456 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:22 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:22.062+0000 7ff115409000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T13:27:22.456 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:22 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:22.159+0000 7ff115409000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T13:27:22.801 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:22 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:22.466+0000 7ff115409000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T13:27:22.801 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:22 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:22.557+0000 7ff115409000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T13:27:23.227 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:22 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:22.968+0000 7ff115409000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T13:27:23.227 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:23.103+0000 7ff115409000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T13:27:23.227 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:23.197+0000 7ff115409000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T13:27:23.551 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:23.312+0000 7ff115409000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T13:27:23.551 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:23.399+0000 7ff115409000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T13:27:24.297 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:23.938+0000 7ff115409000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T13:27:24.551 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:24 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:27:24.297+0000 7ff115409000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T13:27:25.301 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:25 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: [06/May/2024:13:27:25] ENGINE Bus STARTING 2024-05-06T13:27:25.301 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:25 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: [06/May/2024:13:27:25] ENGINE Serving on https://172.21.15.77:7150 2024-05-06T13:27:25.302 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:27:25 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: [06/May/2024:13:27:25] ENGINE Bus STARTED 2024-05-06T13:27:25.362 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: { 2024-05-06T13:27:25.363 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-05-06T13:27:25.363 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "initialized": true 2024-05-06T13:27:25.363 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: } 2024-05-06T13:27:25.712 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:25 smithi077 ceph-mon[22428]: Active manager daemon y restarted 2024-05-06T13:27:25.712 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:25 smithi077 ceph-mon[22428]: Activating manager daemon y 2024-05-06T13:27:25.712 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:25 smithi077 ceph-mon[22428]: osdmap e3: 0 total, 0 up, 0 in 2024-05-06T13:27:25.712 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:25 smithi077 ceph-mon[22428]: mgrmap e10: y(active, starting, since 0.0550974s) 2024-05-06T13:27:25.712 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:25 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:27:25.712 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:25 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T13:27:25.712 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:25 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T13:27:25.713 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:25 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T13:27:25.713 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:25 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T13:27:25.713 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:25 smithi077 ceph-mon[22428]: Manager daemon y is now available 2024-05-06T13:27:25.713 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:25 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:27:25.713 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:25 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:27:25.713 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:25 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:27:25.713 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:25 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:27:25.713 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:25 smithi077 ceph-mon[22428]: [06/May/2024:13:27:25] ENGINE Bus STARTING 2024-05-06T13:27:25.713 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:25 smithi077 ceph-mon[22428]: [06/May/2024:13:27:25] ENGINE Serving on https://172.21.15.77:7150 2024-05-06T13:27:25.713 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:25 smithi077 ceph-mon[22428]: [06/May/2024:13:27:25] ENGINE Bus STARTED 2024-05-06T13:27:25.713 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:25 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:26.018 INFO:teuthology.orchestra.run.smithi077.stderr:mgr epoch 9 is available 2024-05-06T13:27:26.018 INFO:teuthology.orchestra.run.smithi077.stderr:Generating a dashboard self-signed certificate... 2024-05-06T13:27:26.720 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:26 smithi077 ceph-mon[22428]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-06T13:27:26.720 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:26 smithi077 ceph-mon[22428]: mgrmap e11: y(active, since 1.05746s) 2024-05-06T13:27:26.720 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:26 smithi077 ceph-mon[22428]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-06T13:27:26.852 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: Self-signed certificate created 2024-05-06T13:27:27.523 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:27 smithi077 ceph-mon[22428]: mgrmap e12: y(active, since 2s) 2024-05-06T13:27:27.523 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:27 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:27.523 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:27 smithi077 ceph-mon[22428]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:27:27.523 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:27 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:27.523 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:27 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:27.523 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:27 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:27.525 INFO:teuthology.orchestra.run.smithi077.stderr:Creating initial admin user... 2024-05-06T13:27:28.399 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:28 smithi077 ceph-mon[22428]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:27:28.399 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$ilckcKUeOv0wYH7SujC.b.ESH4GzRuvT1S2eXdi/P1TOH/RfEO08W", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1715002048, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-05-06T13:27:29.244 INFO:teuthology.orchestra.run.smithi077.stderr:Fetching dashboard port number... 2024-05-06T13:27:29.477 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:29 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:29.860 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 8443 2024-05-06T13:27:30.739 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:30 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2142070404' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-06T13:27:30.751 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: disabled 2024-05-06T13:27:30.759 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: inactive 2024-05-06T13:27:30.760 INFO:teuthology.orchestra.run.smithi077.stderr:firewalld.service is not enabled 2024-05-06T13:27:30.760 INFO:teuthology.orchestra.run.smithi077.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-05-06T13:27:30.762 INFO:teuthology.orchestra.run.smithi077.stderr:Ceph Dashboard is now available at: 2024-05-06T13:27:30.762 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-05-06T13:27:30.762 INFO:teuthology.orchestra.run.smithi077.stderr: URL: https://smithi077:8443/ 2024-05-06T13:27:30.762 INFO:teuthology.orchestra.run.smithi077.stderr: User: admin 2024-05-06T13:27:30.762 INFO:teuthology.orchestra.run.smithi077.stderr: Password: ko3u2drigw 2024-05-06T13:27:30.762 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-05-06T13:27:30.762 INFO:teuthology.orchestra.run.smithi077.stderr:Enabling autotune for osd_memory_target 2024-05-06T13:27:31.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:31 smithi077 ceph-mon[22428]: mgrmap e13: y(active, since 6s) 2024-05-06T13:27:32.761 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-05-06T13:27:33.376 INFO:teuthology.orchestra.run.smithi077.stderr:You can access the Ceph CLI with: 2024-05-06T13:27:33.376 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-05-06T13:27:33.376 INFO:teuthology.orchestra.run.smithi077.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-05-06T13:27:33.376 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-05-06T13:27:33.376 INFO:teuthology.orchestra.run.smithi077.stderr:Please consider enabling telemetry to help improve Ceph: 2024-05-06T13:27:33.377 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-05-06T13:27:33.377 INFO:teuthology.orchestra.run.smithi077.stderr: ceph telemetry on 2024-05-06T13:27:33.377 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-05-06T13:27:33.377 INFO:teuthology.orchestra.run.smithi077.stderr:For more information see: 2024-05-06T13:27:33.377 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-05-06T13:27:33.377 INFO:teuthology.orchestra.run.smithi077.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-05-06T13:27:33.377 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-05-06T13:27:33.377 INFO:teuthology.orchestra.run.smithi077.stderr:Bootstrap complete. 2024-05-06T13:27:33.410 INFO:tasks.cephadm:Fetching config... 2024-05-06T13:27:33.410 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-05-06T13:27:33.410 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-05-06T13:27:33.427 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-05-06T13:27:33.427 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-05-06T13:27:33.428 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-05-06T13:27:33.483 INFO:tasks.cephadm:Fetching mon keyring... 2024-05-06T13:27:33.483 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-05-06T13:27:33.483 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/mon.a/keyring of=/dev/stdout 2024-05-06T13:27:33.546 INFO:tasks.cephadm:Fetching pub ssh key... 2024-05-06T13:27:33.547 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-05-06T13:27:33.547 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-05-06T13:27:33.602 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-05-06T13:27:33.603 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDorU5zm7WNWvVyPWZQgRYoSPB64avdXM6r90t+56c2qjP+kthcLjmo/lKvwg9BP7ywG2H9zT9hpHoDJDnFhXAiy5tDbVZVBCZuTy66YCHmCA814XKimhPfDmiOT+iI2FTWiJNJUr7nzQ62pAxOl56k7q+eI+JD77+sF3kzr1w3qGvJoAof14VvGeE/y8p62Wj20/pWN7EOXr0S9EhDEG0E4y9pllJ3VaXaZ/Qp0xcBDXFoi0uCclh5LLnAertyUntzvzVW3dNO3D8oKluI1eHtWD/5uPqS1/sZQWEFGvUm6ZgU9UboK4hLmKACKN87RPaVgmlyeJQPHovZnfuhwi/sZHTN2+1zuCWznIS9Fj1E0gZRHhaVifaUM5a/4zP3U2VlXF7Jk3C7/Po7Dq9Gs0pp3XxDUT5F7QGnixehyAS1h3c9ac4+7dTWAlB2ZVY3spPkGbqV3DV/dVbLz3/Y6S/w/o21ZCnbGukDE7iwd+N936izTjOP6JKaJdhLNo3bztM= ceph-20e59cb4-0bac-11ef-bc97-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-06T13:27:33.674 INFO:teuthology.orchestra.run.smithi077.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDorU5zm7WNWvVyPWZQgRYoSPB64avdXM6r90t+56c2qjP+kthcLjmo/lKvwg9BP7ywG2H9zT9hpHoDJDnFhXAiy5tDbVZVBCZuTy66YCHmCA814XKimhPfDmiOT+iI2FTWiJNJUr7nzQ62pAxOl56k7q+eI+JD77+sF3kzr1w3qGvJoAof14VvGeE/y8p62Wj20/pWN7EOXr0S9EhDEG0E4y9pllJ3VaXaZ/Qp0xcBDXFoi0uCclh5LLnAertyUntzvzVW3dNO3D8oKluI1eHtWD/5uPqS1/sZQWEFGvUm6ZgU9UboK4hLmKACKN87RPaVgmlyeJQPHovZnfuhwi/sZHTN2+1zuCWznIS9Fj1E0gZRHhaVifaUM5a/4zP3U2VlXF7Jk3C7/Po7Dq9Gs0pp3XxDUT5F7QGnixehyAS1h3c9ac4+7dTWAlB2ZVY3spPkGbqV3DV/dVbLz3/Y6S/w/o21ZCnbGukDE7iwd+N936izTjOP6JKaJdhLNo3bztM= ceph-20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:27:33.685 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDorU5zm7WNWvVyPWZQgRYoSPB64avdXM6r90t+56c2qjP+kthcLjmo/lKvwg9BP7ywG2H9zT9hpHoDJDnFhXAiy5tDbVZVBCZuTy66YCHmCA814XKimhPfDmiOT+iI2FTWiJNJUr7nzQ62pAxOl56k7q+eI+JD77+sF3kzr1w3qGvJoAof14VvGeE/y8p62Wj20/pWN7EOXr0S9EhDEG0E4y9pllJ3VaXaZ/Qp0xcBDXFoi0uCclh5LLnAertyUntzvzVW3dNO3D8oKluI1eHtWD/5uPqS1/sZQWEFGvUm6ZgU9UboK4hLmKACKN87RPaVgmlyeJQPHovZnfuhwi/sZHTN2+1zuCWznIS9Fj1E0gZRHhaVifaUM5a/4zP3U2VlXF7Jk3C7/Po7Dq9Gs0pp3XxDUT5F7QGnixehyAS1h3c9ac4+7dTWAlB2ZVY3spPkGbqV3DV/dVbLz3/Y6S/w/o21ZCnbGukDE7iwd+N936izTjOP6JKaJdhLNo3bztM= ceph-20e59cb4-0bac-11ef-bc97-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-06T13:27:33.726 INFO:teuthology.orchestra.run.smithi170.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDorU5zm7WNWvVyPWZQgRYoSPB64avdXM6r90t+56c2qjP+kthcLjmo/lKvwg9BP7ywG2H9zT9hpHoDJDnFhXAiy5tDbVZVBCZuTy66YCHmCA814XKimhPfDmiOT+iI2FTWiJNJUr7nzQ62pAxOl56k7q+eI+JD77+sF3kzr1w3qGvJoAof14VvGeE/y8p62Wj20/pWN7EOXr0S9EhDEG0E4y9pllJ3VaXaZ/Qp0xcBDXFoi0uCclh5LLnAertyUntzvzVW3dNO3D8oKluI1eHtWD/5uPqS1/sZQWEFGvUm6ZgU9UboK4hLmKACKN87RPaVgmlyeJQPHovZnfuhwi/sZHTN2+1zuCWznIS9Fj1E0gZRHhaVifaUM5a/4zP3U2VlXF7Jk3C7/Po7Dq9Gs0pp3XxDUT5F7QGnixehyAS1h3c9ac4+7dTWAlB2ZVY3spPkGbqV3DV/dVbLz3/Y6S/w/o21ZCnbGukDE7iwd+N936izTjOP6JKaJdhLNo3bztM= ceph-20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:27:33.738 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-05-06T13:27:33.927 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:33 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/248708125' entity='client.admin' 2024-05-06T13:27:36.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:36 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:36.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:36 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi077", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:27:36.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:36 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:36.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:36 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:27:36.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:36 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:37.534 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-05-06T13:27:37.535 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-05-06T13:27:38.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:37 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/1370591546' entity='client.admin' 2024-05-06T13:27:38.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:37 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:27:38.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:37 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:40.514 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi170 2024-05-06T13:27:40.514 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-05-06T13:27:40.514 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/etc/ceph/ceph.conf 2024-05-06T13:27:40.532 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-05-06T13:27:40.532 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:27:40.590 INFO:tasks.cephadm:Adding host smithi170 to orchestrator... 2024-05-06T13:27:40.590 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph orch host add smithi170 2024-05-06T13:27:41.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:40 smithi077 ceph-mon[22428]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:27:41.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:40 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:41.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:40 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:27:41.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:40 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:27:41.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:40 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:27:41.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:40 smithi077 ceph-mon[22428]: Updating smithi077:/etc/ceph/ceph.conf 2024-05-06T13:27:41.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:40 smithi077 ceph-mon[22428]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:27:41.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:40 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:41.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:40 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:43.265 INFO:teuthology.orchestra.run.smithi077.stdout:Added host 'smithi170' with addr '172.21.15.170' 2024-05-06T13:27:43.784 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph orch host ls --format=json 2024-05-06T13:27:44.373 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:44 smithi077 ceph-mon[22428]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi170", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:27:44.373 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:44 smithi077 ceph-mon[22428]: Deploying cephadm binary to smithi170 2024-05-06T13:27:44.373 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:44 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:44.373 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:44 smithi077 ceph-mon[22428]: Added host smithi170 2024-05-06T13:27:44.373 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:44 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:27:44.373 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:44 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:27:44.373 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:44 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:27:44.373 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:44 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:44.374 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:44 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:45.553 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:27:45.553 INFO:teuthology.orchestra.run.smithi077.stdout:[{"addr": "172.21.15.77", "hostname": "smithi077", "labels": [], "status": ""}, {"addr": "172.21.15.170", "hostname": "smithi170", "labels": [], "status": ""}] 2024-05-06T13:27:46.157 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:46 smithi077 ceph-mon[22428]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:27:46.158 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:46 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:46.181 INFO:tasks.cephadm:Setting crush tunables to default 2024-05-06T13:27:46.181 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph osd crush tunables default 2024-05-06T13:27:47.389 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:47 smithi077 ceph-mon[22428]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-06T13:27:48.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:48 smithi077 ceph-mon[22428]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:27:49.063 INFO:teuthology.orchestra.run.smithi077.stderr:adjusted tunables profile to default 2024-05-06T13:27:49.361 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:49 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/1202206230' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-06T13:27:49.361 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:49 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:49.361 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:49 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:49.361 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:49 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:27:49.361 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:49 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:49.674 INFO:tasks.cephadm:Adding mon.a on smithi077 2024-05-06T13:27:49.674 INFO:tasks.cephadm:Adding mon.c on smithi077 2024-05-06T13:27:49.674 INFO:tasks.cephadm:Adding mon.b on smithi170 2024-05-06T13:27:49.675 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph orch apply mon '3;smithi077:172.21.15.77=a;smithi077:[v2:172.21.15.77:3301,v1:172.21.15.77:6790]=c;smithi170:172.21.15.170=b' 2024-05-06T13:27:50.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:50 smithi077 ceph-mon[22428]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:27:50.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:50 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/1202206230' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-06T13:27:50.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:50 smithi077 ceph-mon[22428]: osdmap e4: 0 total, 0 up, 0 in 2024-05-06T13:27:51.437 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled mon update... 2024-05-06T13:27:52.113 DEBUG:teuthology.orchestra.run.smithi077:mon.c> sudo journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mon.c.service 2024-05-06T13:27:52.115 DEBUG:teuthology.orchestra.run.smithi170:mon.b> sudo journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mon.b.service 2024-05-06T13:27:52.117 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-06T13:27:52.118 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-06T13:27:52.134 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:52 smithi077 ceph-mon[22428]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:27:52.134 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:52 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:52.134 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:52 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:27:52.134 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:52 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:27:52.134 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:52 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:27:52.134 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:52 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:52.134 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:52 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:52.134 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:52 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:27:52.134 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:52 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:27:53.392 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:53 smithi077 ceph-mon[22428]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi077:172.21.15.77=a;smithi077:[v2:172.21.15.77:3301,v1:172.21.15.77:6790]=c;smithi170:172.21.15.170=b", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:27:53.392 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:53 smithi077 ceph-mon[22428]: Saving service mon spec with placement smithi077:172.21.15.77=a;smithi077:[v2:172.21.15.77:3301,v1:172.21.15.77:6790]=c;smithi170:172.21.15.170=b;count:3 2024-05-06T13:27:53.392 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:53 smithi077 ceph-mon[22428]: Updating smithi170:/etc/ceph/ceph.conf 2024-05-06T13:27:53.392 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:53 smithi077 ceph-mon[22428]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:27:53.392 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:53 smithi077 ceph-mon[22428]: Deploying daemon mon.c on smithi077 2024-05-06T13:27:53.980 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-05-06T13:27:53.980 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"20e59cb4-0bac-11ef-bc97-c7b262605968","modified":"2024-05-06T13:26:34.664804Z","created":"2024-05-06T13:26:34.664804Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-06T13:27:53.982 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2024-05-06T13:27:54.320 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[22428]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:27:54.320 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[22428]: from='client.? 172.21.15.170:0/2089604592' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T13:27:54.745 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 systemd[1]: Starting Ceph mon.c for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:27:55.029 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 podman[27147]: 2024-05-06 13:27:54.64886136 +0000 UTC m=+0.018454541 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-05-06T13:27:55.029 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 podman[27147]: 2024-05-06 13:27:54.752205772 +0000 UTC m=+0.121798925 container create 83af336ef762e9b1a830dbf32bb4ece5ad1324f6f1719d4fd97d5f1cac146890 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-c, version=8, io.k8s.display-name=CentOS Stream 8, ceph=True, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, release=754, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, vcs-type=git) 2024-05-06T13:27:55.029 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 podman[27147]: 2024-05-06 13:27:54.946067041 +0000 UTC m=+0.315660206 container init 83af336ef762e9b1a830dbf32bb4ece5ad1324f6f1719d4fd97d5f1cac146890 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-c, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , name=centos-stream, distribution-scope=public, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, release=754, version=8, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8) 2024-05-06T13:27:55.029 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 podman[27147]: 2024-05-06 13:27:54.949165661 +0000 UTC m=+0.318758817 container start 83af336ef762e9b1a830dbf32bb4ece5ad1324f6f1719d4fd97d5f1cac146890 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-c, architecture=x86_64, vcs-type=git, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, ceph=True, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=) 2024-05-06T13:27:55.029 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: set uid:gid to 167:167 (ceph:ceph) 2024-05-06T13:27:55.029 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-05-06T13:27:55.029 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: pidfile_write: ignore empty --pid-file 2024-05-06T13:27:55.029 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: load: jerasure load: lrc 2024-05-06T13:27:55.030 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: RocksDB version: 6.15.5 2024-05-06T13:27:55.030 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-05-06T13:27:55.030 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Compile date Apr 18 2022 2024-05-06T13:27:55.030 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: DB SUMMARY 2024-05-06T13:27:55.030 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: DB Session ID: 7X0O9OG97AJTWQVJ1J24 2024-05-06T13:27:55.030 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: CURRENT file: CURRENT 2024-05-06T13:27:55.030 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: IDENTITY file: IDENTITY 2024-05-06T13:27:55.030 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-05-06T13:27:55.030 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-05-06T13:27:55.030 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-05-06T13:27:55.030 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.error_if_exists: 0 2024-05-06T13:27:55.030 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.create_if_missing: 0 2024-05-06T13:27:55.030 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.paranoid_checks: 1 2024-05-06T13:27:55.030 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-06T13:27:55.031 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.env: 0x55e3f74c6860 2024-05-06T13:27:55.031 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.fs: Posix File System 2024-05-06T13:27:55.031 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.info_log: 0x55e3f8d33dc0 2024-05-06T13:27:55.031 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.max_file_opening_threads: 16 2024-05-06T13:27:55.031 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.statistics: (nil) 2024-05-06T13:27:55.031 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.use_fsync: 0 2024-05-06T13:27:55.032 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.max_log_file_size: 0 2024-05-06T13:27:55.033 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-06T13:27:55.033 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-06T13:27:55.033 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.keep_log_file_num: 1000 2024-05-06T13:27:55.033 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.recycle_log_file_num: 0 2024-05-06T13:27:55.033 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.allow_fallocate: 1 2024-05-06T13:27:55.033 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.allow_mmap_reads: 0 2024-05-06T13:27:55.033 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.allow_mmap_writes: 0 2024-05-06T13:27:55.033 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.use_direct_reads: 0 2024-05-06T13:27:55.033 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-06T13:27:55.033 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.create_missing_column_families: 0 2024-05-06T13:27:55.033 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.db_log_dir: 2024-05-06T13:27:55.033 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-05-06T13:27:55.033 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-06T13:27:55.033 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-06T13:27:55.034 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-06T13:27:55.034 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-06T13:27:55.034 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-06T13:27:55.034 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-06T13:27:55.034 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.advise_random_on_open: 1 2024-05-06T13:27:55.034 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.db_write_buffer_size: 0 2024-05-06T13:27:55.034 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.write_buffer_manager: 0x55e3f8e242d0 2024-05-06T13:27:55.034 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-06T13:27:55.034 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-05-06T13:27:55.034 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-06T13:27:55.034 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-06T13:27:55.034 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.rate_limiter: (nil) 2024-05-06T13:27:55.034 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-06T13:27:55.034 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.wal_recovery_mode: 2 2024-05-06T13:27:55.034 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.enable_thread_tracking: 0 2024-05-06T13:27:55.035 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.enable_pipelined_write: 0 2024-05-06T13:27:55.035 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.unordered_write: 0 2024-05-06T13:27:55.035 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-06T13:27:55.035 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-06T13:27:55.035 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-06T13:27:55.035 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-06T13:27:55.035 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.row_cache: None 2024-05-06T13:27:55.035 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.wal_filter: None 2024-05-06T13:27:55.035 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-06T13:27:55.035 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.allow_ingest_behind: 0 2024-05-06T13:27:55.035 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.preserve_deletes: 0 2024-05-06T13:27:55.035 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.two_write_queues: 0 2024-05-06T13:27:55.035 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.manual_wal_flush: 0 2024-05-06T13:27:55.035 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.atomic_flush: 0 2024-05-06T13:27:55.035 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-06T13:27:55.035 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-06T13:27:55.036 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-06T13:27:55.036 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.log_readahead_size: 0 2024-05-06T13:27:55.036 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-06T13:27:55.036 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.best_efforts_recovery: 0 2024-05-06T13:27:55.036 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-06T13:27:55.036 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-06T13:27:55.036 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.allow_data_in_errors: 0 2024-05-06T13:27:55.036 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.db_host_id: __hostname__ 2024-05-06T13:27:55.036 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.max_background_jobs: 2 2024-05-06T13:27:55.036 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.max_background_compactions: -1 2024-05-06T13:27:55.037 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.max_subcompactions: 1 2024-05-06T13:27:55.037 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-06T13:27:55.037 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-06T13:27:55.037 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-06T13:27:55.037 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.max_total_wal_size: 0 2024-05-06T13:27:55.037 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-06T13:27:55.037 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-06T13:27:55.037 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-06T13:27:55.037 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-06T13:27:55.037 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.max_open_files: -1 2024-05-06T13:27:55.037 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.bytes_per_sync: 0 2024-05-06T13:27:55.037 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-06T13:27:55.037 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-06T13:27:55.037 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.compaction_readahead_size: 0 2024-05-06T13:27:55.037 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.max_background_flushes: -1 2024-05-06T13:27:55.037 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Compression algorithms supported: 2024-05-06T13:27:55.038 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-06T13:27:55.038 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: kZSTD supported: 0 2024-05-06T13:27:55.038 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: kXpressCompression supported: 0 2024-05-06T13:27:55.038 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: kLZ4HCCompression supported: 1 2024-05-06T13:27:55.038 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: kLZ4Compression supported: 1 2024-05-06T13:27:55.038 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: kBZip2Compression supported: 0 2024-05-06T13:27:55.038 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: kZlibCompression supported: 1 2024-05-06T13:27:55.038 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: kSnappyCompression supported: 1 2024-05-06T13:27:55.038 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-06T13:27:55.038 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-05-06T13:27:55.038 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-05-06T13:27:55.038 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-06T13:27:55.038 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.merge_operator: 2024-05-06T13:27:55.038 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.compaction_filter: None 2024-05-06T13:27:55.039 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.compaction_filter_factory: None 2024-05-06T13:27:55.039 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.sst_partitioner_factory: None 2024-05-06T13:27:55.039 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-06T13:27:55.039 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-06T13:27:55.039 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e3f8cffd00) 2024-05-06T13:27:55.039 INFO:journalctl@ceph.mon.c.smithi077.stdout: cache_index_and_filter_blocks: 1 2024-05-06T13:27:55.039 INFO:journalctl@ceph.mon.c.smithi077.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-06T13:27:55.039 INFO:journalctl@ceph.mon.c.smithi077.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-06T13:27:55.039 INFO:journalctl@ceph.mon.c.smithi077.stdout: pin_top_level_index_and_filter: 1 2024-05-06T13:27:55.039 INFO:journalctl@ceph.mon.c.smithi077.stdout: index_type: 0 2024-05-06T13:27:55.039 INFO:journalctl@ceph.mon.c.smithi077.stdout: data_block_index_type: 0 2024-05-06T13:27:55.039 INFO:journalctl@ceph.mon.c.smithi077.stdout: index_shortening: 1 2024-05-06T13:27:55.039 INFO:journalctl@ceph.mon.c.smithi077.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-06T13:27:55.039 INFO:journalctl@ceph.mon.c.smithi077.stdout: hash_index_allow_collision: 1 2024-05-06T13:27:55.039 INFO:journalctl@ceph.mon.c.smithi077.stdout: checksum: 1 2024-05-06T13:27:55.040 INFO:journalctl@ceph.mon.c.smithi077.stdout: no_block_cache: 0 2024-05-06T13:27:55.040 INFO:journalctl@ceph.mon.c.smithi077.stdout: block_cache: 0x55e3f8d6a170 2024-05-06T13:27:55.040 INFO:journalctl@ceph.mon.c.smithi077.stdout: block_cache_name: BinnedLRUCache 2024-05-06T13:27:55.040 INFO:journalctl@ceph.mon.c.smithi077.stdout: block_cache_options: 2024-05-06T13:27:55.040 INFO:journalctl@ceph.mon.c.smithi077.stdout: capacity : 536870912 2024-05-06T13:27:55.040 INFO:journalctl@ceph.mon.c.smithi077.stdout: num_shard_bits : 4 2024-05-06T13:27:55.040 INFO:journalctl@ceph.mon.c.smithi077.stdout: strict_capacity_limit : 0 2024-05-06T13:27:55.040 INFO:journalctl@ceph.mon.c.smithi077.stdout: high_pri_pool_ratio: 0.000 2024-05-06T13:27:55.040 INFO:journalctl@ceph.mon.c.smithi077.stdout: block_cache_compressed: (nil) 2024-05-06T13:27:55.040 INFO:journalctl@ceph.mon.c.smithi077.stdout: persistent_cache: (nil) 2024-05-06T13:27:55.040 INFO:journalctl@ceph.mon.c.smithi077.stdout: block_size: 4096 2024-05-06T13:27:55.040 INFO:journalctl@ceph.mon.c.smithi077.stdout: block_size_deviation: 10 2024-05-06T13:27:55.040 INFO:journalctl@ceph.mon.c.smithi077.stdout: block_restart_interval: 16 2024-05-06T13:27:55.040 INFO:journalctl@ceph.mon.c.smithi077.stdout: index_block_restart_interval: 1 2024-05-06T13:27:55.040 INFO:journalctl@ceph.mon.c.smithi077.stdout: metadata_block_size: 4096 2024-05-06T13:27:55.040 INFO:journalctl@ceph.mon.c.smithi077.stdout: partition_filters: 0 2024-05-06T13:27:55.041 INFO:journalctl@ceph.mon.c.smithi077.stdout: use_delta_encoding: 1 2024-05-06T13:27:55.041 INFO:journalctl@ceph.mon.c.smithi077.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-05-06T13:27:55.041 INFO:journalctl@ceph.mon.c.smithi077.stdout: whole_key_filtering: 1 2024-05-06T13:27:55.041 INFO:journalctl@ceph.mon.c.smithi077.stdout: verify_compression: 0 2024-05-06T13:27:55.041 INFO:journalctl@ceph.mon.c.smithi077.stdout: read_amp_bytes_per_bit: 0 2024-05-06T13:27:55.041 INFO:journalctl@ceph.mon.c.smithi077.stdout: format_version: 4 2024-05-06T13:27:55.041 INFO:journalctl@ceph.mon.c.smithi077.stdout: enable_index_compression: 1 2024-05-06T13:27:55.041 INFO:journalctl@ceph.mon.c.smithi077.stdout: block_align: 0 2024-05-06T13:27:55.041 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.write_buffer_size: 33554432 2024-05-06T13:27:55.041 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.max_write_buffer_number: 2 2024-05-06T13:27:55.041 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.compression: NoCompression 2024-05-06T13:27:55.041 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.bottommost_compression: Disabled 2024-05-06T13:27:55.041 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.prefix_extractor: nullptr 2024-05-06T13:27:55.041 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-06T13:27:55.041 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.num_levels: 7 2024-05-06T13:27:55.042 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-06T13:27:55.042 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-06T13:27:55.042 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-06T13:27:55.042 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-06T13:27:55.042 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-06T13:27:55.042 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-06T13:27:55.042 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-06T13:27:55.042 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-06T13:27:55.042 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-06T13:27:55.042 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-06T13:27:55.042 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-06T13:27:55.042 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.compression_opts.level: 32767 2024-05-06T13:27:55.042 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.compression_opts.strategy: 0 2024-05-06T13:27:55.042 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-06T13:27:55.042 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-06T13:27:55.043 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-06T13:27:55.043 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.compression_opts.enabled: false 2024-05-06T13:27:55.043 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-06T13:27:55.043 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-06T13:27:55.043 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-06T13:27:55.043 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.target_file_size_base: 67108864 2024-05-06T13:27:55.043 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-06T13:27:55.043 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-06T13:27:55.043 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-06T13:27:55.043 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-06T13:27:55.043 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-06T13:27:55.043 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-06T13:27:55.043 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-06T13:27:55.043 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-06T13:27:55.043 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-06T13:27:55.044 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-06T13:27:55.044 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-06T13:27:55.044 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-06T13:27:55.044 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-06T13:27:55.044 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.arena_block_size: 4194304 2024-05-06T13:27:55.044 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-06T13:27:55.044 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-06T13:27:55.044 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-05-06T13:27:55.044 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.disable_auto_compactions: 0 2024-05-06T13:27:55.044 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-06T13:27:55.044 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-06T13:27:55.044 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-06T13:27:55.044 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-06T13:27:55.044 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-06T13:27:55.044 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-06T13:27:55.045 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-06T13:27:55.045 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-06T13:27:55.045 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-06T13:27:55.045 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-06T13:27:55.045 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.table_properties_collectors: 2024-05-06T13:27:55.045 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.inplace_update_support: 0 2024-05-06T13:27:55.045 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-06T13:27:55.045 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-06T13:27:55.045 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-06T13:27:55.045 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-06T13:27:55.045 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.bloom_locality: 0 2024-05-06T13:27:55.045 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.max_successive_merges: 0 2024-05-06T13:27:55.045 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-06T13:27:55.045 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.paranoid_file_checks: 0 2024-05-06T13:27:55.045 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.force_consistency_checks: 1 2024-05-06T13:27:55.045 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.report_bg_io_stats: 0 2024-05-06T13:27:55.046 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.ttl: 2592000 2024-05-06T13:27:55.046 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-06T13:27:55.046 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.enable_blob_files: false 2024-05-06T13:27:55.046 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.min_blob_size: 0 2024-05-06T13:27:55.046 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.blob_file_size: 268435456 2024-05-06T13:27:55.046 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-06T13:27:55.046 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-06T13:27:55.046 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-06T13:27:55.046 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-06T13:27:55.046 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-05-06T13:27:55.046 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-05-06T13:27:55.046 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715002074974935, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-05-06T13:27:55.046 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-05-06T13:27:55.046 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-05-06T13:27:55.046 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715002074975233, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1715002074, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "1c965386-f64c-4736-a27d-31c3b9e5d48e", "db_session_id": "7X0O9OG97AJTWQVJ1J24"}} 2024-05-06T13:27:55.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-05-06T13:27:55.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715002074975542, "job": 1, "event": "recovery_finished"} 2024-05-06T13:27:55.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-06T13:27:55.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55e3f8d50a80 2024-05-06T13:27:55.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: DB pointer 0x55e3f8d60000 2024-05-06T13:27:55.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-05-06T13:27:55.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-05-06T13:27:55.047 INFO:journalctl@ceph.mon.c.smithi077.stdout: ** DB Stats ** 2024-05-06T13:27:55.047 INFO:journalctl@ceph.mon.c.smithi077.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T13:27:55.047 INFO:journalctl@ceph.mon.c.smithi077.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-06T13:27:55.047 INFO:journalctl@ceph.mon.c.smithi077.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-06T13:27:55.047 INFO:journalctl@ceph.mon.c.smithi077.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-06T13:27:55.047 INFO:journalctl@ceph.mon.c.smithi077.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-06T13:27:55.047 INFO:journalctl@ceph.mon.c.smithi077.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-05-06T13:27:55.047 INFO:journalctl@ceph.mon.c.smithi077.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-06T13:27:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout: 2024-05-06T13:27:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout: ** Compaction Stats [default] ** 2024-05-06T13:27:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-06T13:27:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-06T13:27:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-05-06T13:27:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-05-06T13:27:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-05-06T13:27:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout: 2024-05-06T13:27:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout: ** Compaction Stats [default] ** 2024-05-06T13:27:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-06T13:27:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-06T13:27:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-05-06T13:27:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T13:27:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-06T13:27:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-06T13:27:55.049 INFO:journalctl@ceph.mon.c.smithi077.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-06T13:27:55.049 INFO:journalctl@ceph.mon.c.smithi077.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-06T13:27:55.049 INFO:journalctl@ceph.mon.c.smithi077.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-06T13:27:55.049 INFO:journalctl@ceph.mon.c.smithi077.stdout: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T13:27:55.049 INFO:journalctl@ceph.mon.c.smithi077.stdout: Interval compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T13:27:55.049 INFO:journalctl@ceph.mon.c.smithi077.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-06T13:27:55.049 INFO:journalctl@ceph.mon.c.smithi077.stdout: 2024-05-06T13:27:55.049 INFO:journalctl@ceph.mon.c.smithi077.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-06T13:27:55.049 INFO:journalctl@ceph.mon.c.smithi077.stdout: 2024-05-06T13:27:55.049 INFO:journalctl@ceph.mon.c.smithi077.stdout: ** Compaction Stats [default] ** 2024-05-06T13:27:55.049 INFO:journalctl@ceph.mon.c.smithi077.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-06T13:27:55.049 INFO:journalctl@ceph.mon.c.smithi077.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-06T13:27:55.049 INFO:journalctl@ceph.mon.c.smithi077.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-05-06T13:27:55.049 INFO:journalctl@ceph.mon.c.smithi077.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-05-06T13:27:55.049 INFO:journalctl@ceph.mon.c.smithi077.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-05-06T13:27:55.050 INFO:journalctl@ceph.mon.c.smithi077.stdout: 2024-05-06T13:27:55.050 INFO:journalctl@ceph.mon.c.smithi077.stdout: ** Compaction Stats [default] ** 2024-05-06T13:27:55.050 INFO:journalctl@ceph.mon.c.smithi077.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-06T13:27:55.050 INFO:journalctl@ceph.mon.c.smithi077.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-06T13:27:55.050 INFO:journalctl@ceph.mon.c.smithi077.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-05-06T13:27:55.050 INFO:journalctl@ceph.mon.c.smithi077.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T13:27:55.050 INFO:journalctl@ceph.mon.c.smithi077.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-06T13:27:55.050 INFO:journalctl@ceph.mon.c.smithi077.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-06T13:27:55.050 INFO:journalctl@ceph.mon.c.smithi077.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-06T13:27:55.050 INFO:journalctl@ceph.mon.c.smithi077.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-06T13:27:55.050 INFO:journalctl@ceph.mon.c.smithi077.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-06T13:27:55.050 INFO:journalctl@ceph.mon.c.smithi077.stdout: Cumulative compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T13:27:55.050 INFO:journalctl@ceph.mon.c.smithi077.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T13:27:55.050 INFO:journalctl@ceph.mon.c.smithi077.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-06T13:27:55.051 INFO:journalctl@ceph.mon.c.smithi077.stdout: 2024-05-06T13:27:55.051 INFO:journalctl@ceph.mon.c.smithi077.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-06T13:27:55.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-05-06T13:27:55.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: using public_addrv [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] 2024-05-06T13:27:55.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: starting mon.c rank -1 at public addrs [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] at bind addrs [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:27:55.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mon.c@-1(???) e0 preinit fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:27:55.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mon.c@-1(synchronizing).mds e1 new map 2024-05-06T13:27:55.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mon.c@-1(synchronizing).mds e1 print_map 2024-05-06T13:27:55.051 INFO:journalctl@ceph.mon.c.smithi077.stdout: e1 2024-05-06T13:27:55.051 INFO:journalctl@ceph.mon.c.smithi077.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-06T13:27:55.051 INFO:journalctl@ceph.mon.c.smithi077.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-06T13:27:55.051 INFO:journalctl@ceph.mon.c.smithi077.stdout: legacy client fscid: -1 2024-05-06T13:27:55.051 INFO:journalctl@ceph.mon.c.smithi077.stdout: 2024-05-06T13:27:55.051 INFO:journalctl@ceph.mon.c.smithi077.stdout: No filesystems configured 2024-05-06T13:27:55.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-06T13:27:55.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-06T13:27:55.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-06T13:27:55.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-06T13:27:55.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-06T13:27:55.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-06T13:27:55.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-06T13:27:55.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-06T13:27:55.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-06T13:27:55.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-06T13:27:55.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mkfs 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:27:55.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-06T13:27:55.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-06T13:27:55.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: monmap e1: 1 mons at {a=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0]} 2024-05-06T13:27:55.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: fsmap 2024-05-06T13:27:55.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: osdmap e1: 0 total, 0 up, 0 in 2024-05-06T13:27:55.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mgrmap e1: no daemons active 2024-05-06T13:27:55.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/4170924274' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-06T13:27:55.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3249644945' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-06T13:27:55.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3249644945' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-06T13:27:55.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/1762607703' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:27:55.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-06T13:27:55.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: monmap e1: 1 mons at {a=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0]} 2024-05-06T13:27:55.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: fsmap 2024-05-06T13:27:55.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: osdmap e1: 0 total, 0 up, 0 in 2024-05-06T13:27:55.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mgrmap e1: no daemons active 2024-05-06T13:27:55.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3776430626' entity='client.admin' 2024-05-06T13:27:55.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/889258358' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-06T13:27:55.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/1857539211' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-06T13:27:55.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: Activating manager daemon y 2024-05-06T13:27:55.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mgrmap e2: y(active, starting, since 0.00303351s) 2024-05-06T13:27:55.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14100 172.21.15.77:0/4000433697' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T13:27:55.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14100 172.21.15.77:0/4000433697' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T13:27:55.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14100 172.21.15.77:0/4000433697' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T13:27:55.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14100 172.21.15.77:0/4000433697' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:27:55.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14100 172.21.15.77:0/4000433697' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T13:27:55.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: Manager daemon y is now available 2024-05-06T13:27:55.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14100 172.21.15.77:0/4000433697' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:27:55.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14100 172.21.15.77:0/4000433697' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:27:55.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14100 172.21.15.77:0/4000433697' entity='mgr.y' 2024-05-06T13:27:55.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14100 172.21.15.77:0/4000433697' entity='mgr.y' 2024-05-06T13:27:55.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14100 172.21.15.77:0/4000433697' entity='mgr.y' 2024-05-06T13:27:55.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mgrmap e3: y(active, since 1.00766s) 2024-05-06T13:27:55.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/4081279330' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-06T13:27:55.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mgrmap e4: y(active, since 2s) 2024-05-06T13:27:55.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/483883626' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-06T13:27:55.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/483883626' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-06T13:27:55.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3264000791' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-06T13:27:55.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3264000791' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-06T13:27:55.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mgrmap e5: y(active, since 4s) 2024-05-06T13:27:55.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/4146590142' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-06T13:27:55.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: Active manager daemon y restarted 2024-05-06T13:27:55.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: Activating manager daemon y 2024-05-06T13:27:55.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: osdmap e2: 0 total, 0 up, 0 in 2024-05-06T13:27:55.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mgrmap e6: y(active, starting, since 0.0524327s) 2024-05-06T13:27:55.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:27:55.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T13:27:55.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T13:27:55.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T13:27:55.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T13:27:55.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: Manager daemon y is now available 2024-05-06T13:27:55.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' 2024-05-06T13:27:55.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' 2024-05-06T13:27:55.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:27:55.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:27:55.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:27:55.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:27:55.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:27:55.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: [06/May/2024:13:27:03] ENGINE Bus STARTING 2024-05-06T13:27:55.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-06T13:27:55.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mgrmap e7: y(active, since 1.05747s) 2024-05-06T13:27:55.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-06T13:27:55.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: [06/May/2024:13:27:03] ENGINE Serving on https://172.21.15.77:7150 2024-05-06T13:27:55.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: [06/May/2024:13:27:03] ENGINE Bus STARTED 2024-05-06T13:27:55.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' 2024-05-06T13:27:55.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:27:55.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mgrmap e8: y(active, since 2s) 2024-05-06T13:27:55.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' 2024-05-06T13:27:55.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:27:55.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:27:55.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:27:55.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' 2024-05-06T13:27:55.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' 2024-05-06T13:27:55.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:27:55.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: Generating ssh key... 2024-05-06T13:27:55.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:27:55.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi077", "addr": "172.21.15.77", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:27:55.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: Deploying cephadm binary to smithi077 2024-05-06T13:27:55.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' 2024-05-06T13:27:55.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: Added host smithi077 2024-05-06T13:27:55.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:27:55.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:27:55.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: Saving service mon spec with placement count:5 2024-05-06T13:27:55.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' 2024-05-06T13:27:55.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:27:55.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: Saving service mgr spec with placement count:2 2024-05-06T13:27:55.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' 2024-05-06T13:27:55.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' 2024-05-06T13:27:55.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' 2024-05-06T13:27:55.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2721465418' entity='client.admin' 2024-05-06T13:27:55.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3709216103' entity='client.admin' 2024-05-06T13:27:55.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' 2024-05-06T13:27:55.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3574573116' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-06T13:27:55.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14118 172.21.15.77:0/3293045602' entity='mgr.y' 2024-05-06T13:27:55.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3574573116' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-06T13:27:55.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mgrmap e9: y(active, since 16s) 2024-05-06T13:27:55.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2509053377' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-06T13:27:55.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: Active manager daemon y restarted 2024-05-06T13:27:55.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: Activating manager daemon y 2024-05-06T13:27:55.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: osdmap e3: 0 total, 0 up, 0 in 2024-05-06T13:27:55.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mgrmap e10: y(active, starting, since 0.0550974s) 2024-05-06T13:27:55.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:27:55.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T13:27:55.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T13:27:55.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T13:27:55.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T13:27:55.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: Manager daemon y is now available 2024-05-06T13:27:55.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:27:55.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:27:55.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:27:55.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:27:55.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: [06/May/2024:13:27:25] ENGINE Bus STARTING 2024-05-06T13:27:55.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: [06/May/2024:13:27:25] ENGINE Serving on https://172.21.15.77:7150 2024-05-06T13:27:55.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: [06/May/2024:13:27:25] ENGINE Bus STARTED 2024-05-06T13:27:55.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:55.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-06T13:27:55.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mgrmap e11: y(active, since 1.05746s) 2024-05-06T13:27:55.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-06T13:27:55.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mgrmap e12: y(active, since 2s) 2024-05-06T13:27:55.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:55.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:27:55.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:55.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:55.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:55.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:27:55.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:55.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2142070404' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-06T13:27:55.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mgrmap e13: y(active, since 6s) 2024-05-06T13:27:55.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/248708125' entity='client.admin' 2024-05-06T13:27:55.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:55.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi077", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:27:55.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:55.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:27:55.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:55.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/1370591546' entity='client.admin' 2024-05-06T13:27:55.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:27:55.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:55.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:27:55.063 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:55.063 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:27:55.063 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:27:55.063 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:27:55.063 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: Updating smithi077:/etc/ceph/ceph.conf 2024-05-06T13:27:55.063 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:27:55.063 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:55.063 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:55.063 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi170", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:27:55.063 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: Deploying cephadm binary to smithi170 2024-05-06T13:27:55.063 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:55.063 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: Added host smithi170 2024-05-06T13:27:55.063 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:27:55.063 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:27:55.064 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:27:55.064 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:55.064 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:55.064 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:27:55.064 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:55.064 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-06T13:27:55.064 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:27:55.064 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/1202206230' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-06T13:27:55.064 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:55.064 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:55.064 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:27:55.064 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:55.064 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:27:55.065 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/1202206230' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-06T13:27:55.065 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: osdmap e4: 0 total, 0 up, 0 in 2024-05-06T13:27:55.065 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:27:55.065 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:55.065 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:27:55.065 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:27:55.065 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:27:55.065 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:55.065 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:27:55.065 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:27:55.065 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:27:55.065 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi077:172.21.15.77=a;smithi077:[v2:172.21.15.77:3301,v1:172.21.15.77:6790]=c;smithi170:172.21.15.170=b", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:27:55.065 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: Saving service mon spec with placement smithi077:172.21.15.77=a;smithi077:[v2:172.21.15.77:3301,v1:172.21.15.77:6790]=c;smithi170:172.21.15.170=b;count:3 2024-05-06T13:27:55.065 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: Updating smithi170:/etc/ceph/ceph.conf 2024-05-06T13:27:55.066 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:27:55.066 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: Deploying daemon mon.c on smithi077 2024-05-06T13:27:55.066 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:27:55.066 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.170:0/2089604592' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T13:27:55.066 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:54 smithi077 ceph-mon[27166]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-05-06T13:27:55.066 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:55 smithi077 bash[27147]: 83af336ef762e9b1a830dbf32bb4ece5ad1324f6f1719d4fd97d5f1cac146890 2024-05-06T13:27:55.066 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:27:55 smithi077 systemd[1]: Started Ceph mon.c for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:27:55.540 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-06T13:27:55.540 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-06T13:28:00.455 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-05-06T13:28:00.455 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":2,"fsid":"20e59cb4-0bac-11ef-bc97-c7b262605968","modified":"2024-05-06T13:27:54.993699Z","created":"2024-05-06T13:26:34.664804Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3301","nonce":0},{"type":"v1","addr":"172.21.15.77:6790","nonce":0}]},"addr":"172.21.15.77:6790/0","public_addr":"172.21.15.77:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-06T13:28:00.457 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 2 2024-05-06T13:28:00.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[22428]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:00.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:28:00.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[22428]: mon.a calling monitor election 2024-05-06T13:28:00.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:28:00.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:28:00.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[22428]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:00.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:28:00.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[22428]: mon.c calling monitor election 2024-05-06T13:28:00.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:28:00.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[22428]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:00.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:28:00.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:28:00.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[22428]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-06T13:28:00.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[22428]: monmap e2: 2 mons at {a=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0],c=[v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0]} 2024-05-06T13:28:00.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[22428]: fsmap 2024-05-06T13:28:00.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[22428]: osdmap e4: 0 total, 0 up, 0 in 2024-05-06T13:28:00.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[22428]: mgrmap e13: y(active, since 35s) 2024-05-06T13:28:00.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[22428]: overall HEALTH_OK 2024-05-06T13:28:00.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:00.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:28:00.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:00.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[27166]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:00.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:28:00.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[27166]: mon.a calling monitor election 2024-05-06T13:28:00.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:28:00.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:28:00.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[27166]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:00.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:28:00.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[27166]: mon.c calling monitor election 2024-05-06T13:28:00.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:28:00.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[27166]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:00.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:28:00.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:28:00.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[27166]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-06T13:28:00.555 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[27166]: monmap e2: 2 mons at {a=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0],c=[v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0]} 2024-05-06T13:28:00.555 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[27166]: fsmap 2024-05-06T13:28:00.555 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[27166]: osdmap e4: 0 total, 0 up, 0 in 2024-05-06T13:28:00.555 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[27166]: mgrmap e13: y(active, since 35s) 2024-05-06T13:28:00.555 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[27166]: overall HEALTH_OK 2024-05-06T13:28:00.555 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:00.555 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:28:00.555 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:00 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:01.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:01 smithi077 ceph-mon[22428]: Deploying daemon mon.b on smithi170 2024-05-06T13:28:01.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:01 smithi077 ceph-mon[22428]: from='client.? 172.21.15.170:0/864607528' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T13:28:01.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:01 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:28:01.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:01 smithi077 ceph-mon[27166]: Deploying daemon mon.b on smithi170 2024-05-06T13:28:01.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:01 smithi077 ceph-mon[27166]: from='client.? 172.21.15.170:0/864607528' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T13:28:01.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:01 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:28:02.030 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-06T13:28:02.030 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-06T13:28:02.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:02 smithi077 ceph-mon[22428]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:02.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:02 smithi077 ceph-mon[27166]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:04.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:04 smithi077 ceph-mon[22428]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:04.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:04 smithi077 ceph-mon[27166]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:04.638 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-05-06T13:28:04.638 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":2,"fsid":"20e59cb4-0bac-11ef-bc97-c7b262605968","modified":"2024-05-06T13:27:54.993699Z","created":"2024-05-06T13:26:34.664804Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3301","nonce":0},{"type":"v1","addr":"172.21.15.77:6790","nonce":0}]},"addr":"172.21.15.77:6790/0","public_addr":"172.21.15.77:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-06T13:28:04.639 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 2 2024-05-06T13:28:05.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 systemd[1]: Starting Ceph mon.b for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:28:05.801 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:05 smithi077 ceph-mon[27166]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:05.801 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:05 smithi077 ceph-mon[27166]: from='client.? 172.21.15.170:0/905814282' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T13:28:05.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:05 smithi077 ceph-mon[22428]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:05.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:05 smithi077 ceph-mon[22428]: from='client.? 172.21.15.170:0/905814282' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T13:28:05.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 podman[23253]: 2024-05-06 13:28:05.493139348 +0000 UTC m=+0.030496229 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-05-06T13:28:05.835 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 podman[23253]: 2024-05-06 13:28:05.594398957 +0000 UTC m=+0.131755873 container create 24be6fce9d28a1fec954db1eb8cf7b5583295d6e779e9ae383c08f400988d439 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-b, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, com.redhat.component=centos-stream-container, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, GIT_CLEAN=True, name=centos-stream, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., RELEASE=HEAD, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754) 2024-05-06T13:28:05.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 podman[23253]: 2024-05-06 13:28:05.791158018 +0000 UTC m=+0.328514888 container init 24be6fce9d28a1fec954db1eb8cf7b5583295d6e779e9ae383c08f400988d439 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-b, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, name=centos-stream, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , version=8, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.k8s.display-name=CentOS Stream 8, vcs-type=git, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, release=754, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, architecture=x86_64, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-05-06T13:28:05.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 podman[23253]: 2024-05-06 13:28:05.794270086 +0000 UTC m=+0.331626956 container start 24be6fce9d28a1fec954db1eb8cf7b5583295d6e779e9ae383c08f400988d439 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-b, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, io.buildah.version=1.19.8, distribution-scope=public, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, release=754, GIT_BRANCH=HEAD, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, architecture=x86_64, version=8, vcs-type=git) 2024-05-06T13:28:05.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: set uid:gid to 167:167 (ceph:ceph) 2024-05-06T13:28:05.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-05-06T13:28:05.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: pidfile_write: ignore empty --pid-file 2024-05-06T13:28:05.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: load: jerasure load: lrc 2024-05-06T13:28:05.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: RocksDB version: 6.15.5 2024-05-06T13:28:05.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-05-06T13:28:05.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Compile date Apr 18 2022 2024-05-06T13:28:05.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: DB SUMMARY 2024-05-06T13:28:05.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: DB Session ID: SX7O0OYYAYK419P4E4ZY 2024-05-06T13:28:05.836 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: CURRENT file: CURRENT 2024-05-06T13:28:05.837 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: IDENTITY file: IDENTITY 2024-05-06T13:28:05.837 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-05-06T13:28:05.837 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-05-06T13:28:05.837 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-05-06T13:28:05.837 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.error_if_exists: 0 2024-05-06T13:28:05.837 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.create_if_missing: 0 2024-05-06T13:28:05.837 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.paranoid_checks: 1 2024-05-06T13:28:05.837 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-06T13:28:05.837 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.env: 0x55bc2cac6860 2024-05-06T13:28:05.837 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.fs: Posix File System 2024-05-06T13:28:05.837 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.info_log: 0x55bc2dcfbdc0 2024-05-06T13:28:05.837 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.max_file_opening_threads: 16 2024-05-06T13:28:05.837 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.statistics: (nil) 2024-05-06T13:28:05.838 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.use_fsync: 0 2024-05-06T13:28:05.839 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.max_log_file_size: 0 2024-05-06T13:28:05.839 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-06T13:28:05.839 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-06T13:28:05.839 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.keep_log_file_num: 1000 2024-05-06T13:28:05.839 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.recycle_log_file_num: 0 2024-05-06T13:28:05.839 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.allow_fallocate: 1 2024-05-06T13:28:05.840 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.allow_mmap_reads: 0 2024-05-06T13:28:05.840 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.allow_mmap_writes: 0 2024-05-06T13:28:05.840 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.use_direct_reads: 0 2024-05-06T13:28:05.840 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-06T13:28:05.840 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.create_missing_column_families: 0 2024-05-06T13:28:05.840 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.db_log_dir: 2024-05-06T13:28:05.840 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-05-06T13:28:05.840 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-06T13:28:05.840 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-06T13:28:05.840 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-06T13:28:05.840 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-06T13:28:05.841 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-06T13:28:05.841 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-06T13:28:05.841 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.advise_random_on_open: 1 2024-05-06T13:28:05.841 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.db_write_buffer_size: 0 2024-05-06T13:28:05.841 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.write_buffer_manager: 0x55bc2ddec240 2024-05-06T13:28:05.841 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-06T13:28:05.841 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-05-06T13:28:05.841 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-06T13:28:05.841 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-06T13:28:05.841 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.rate_limiter: (nil) 2024-05-06T13:28:05.841 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-06T13:28:05.841 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.wal_recovery_mode: 2 2024-05-06T13:28:05.841 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.enable_thread_tracking: 0 2024-05-06T13:28:05.842 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.enable_pipelined_write: 0 2024-05-06T13:28:05.842 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.unordered_write: 0 2024-05-06T13:28:05.842 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-06T13:28:05.842 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-06T13:28:05.842 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-06T13:28:05.842 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-06T13:28:05.842 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.row_cache: None 2024-05-06T13:28:05.842 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.wal_filter: None 2024-05-06T13:28:05.842 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-06T13:28:05.842 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.allow_ingest_behind: 0 2024-05-06T13:28:05.842 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.preserve_deletes: 0 2024-05-06T13:28:05.842 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.two_write_queues: 0 2024-05-06T13:28:05.843 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.manual_wal_flush: 0 2024-05-06T13:28:05.843 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.atomic_flush: 0 2024-05-06T13:28:05.843 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-06T13:28:05.843 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-06T13:28:05.843 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-06T13:28:05.843 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.log_readahead_size: 0 2024-05-06T13:28:05.844 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-06T13:28:05.844 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.best_efforts_recovery: 0 2024-05-06T13:28:05.844 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-06T13:28:05.844 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-06T13:28:05.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.allow_data_in_errors: 0 2024-05-06T13:28:05.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.db_host_id: __hostname__ 2024-05-06T13:28:05.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.max_background_jobs: 2 2024-05-06T13:28:05.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.max_background_compactions: -1 2024-05-06T13:28:05.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.max_subcompactions: 1 2024-05-06T13:28:05.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-06T13:28:05.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-06T13:28:05.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-06T13:28:05.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.max_total_wal_size: 0 2024-05-06T13:28:05.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-06T13:28:05.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-06T13:28:05.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-06T13:28:05.845 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-06T13:28:05.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.max_open_files: -1 2024-05-06T13:28:05.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.bytes_per_sync: 0 2024-05-06T13:28:05.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-06T13:28:05.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-06T13:28:05.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.compaction_readahead_size: 0 2024-05-06T13:28:05.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.max_background_flushes: -1 2024-05-06T13:28:05.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Compression algorithms supported: 2024-05-06T13:28:05.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-06T13:28:05.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: kZSTD supported: 0 2024-05-06T13:28:05.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: kXpressCompression supported: 0 2024-05-06T13:28:05.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: kLZ4HCCompression supported: 1 2024-05-06T13:28:05.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: kLZ4Compression supported: 1 2024-05-06T13:28:05.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: kBZip2Compression supported: 0 2024-05-06T13:28:05.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: kZlibCompression supported: 1 2024-05-06T13:28:05.846 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: kSnappyCompression supported: 1 2024-05-06T13:28:05.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-06T13:28:05.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2024-05-06T13:28:05.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-05-06T13:28:05.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-06T13:28:05.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.merge_operator: 2024-05-06T13:28:05.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.compaction_filter: None 2024-05-06T13:28:05.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.compaction_filter_factory: None 2024-05-06T13:28:05.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.sst_partitioner_factory: None 2024-05-06T13:28:05.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-06T13:28:05.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-06T13:28:05.847 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55bc2dcc7d00) 2024-05-06T13:28:05.847 INFO:journalctl@ceph.mon.b.smithi170.stdout: cache_index_and_filter_blocks: 1 2024-05-06T13:28:05.847 INFO:journalctl@ceph.mon.b.smithi170.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-06T13:28:05.847 INFO:journalctl@ceph.mon.b.smithi170.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-06T13:28:05.847 INFO:journalctl@ceph.mon.b.smithi170.stdout: pin_top_level_index_and_filter: 1 2024-05-06T13:28:05.848 INFO:journalctl@ceph.mon.b.smithi170.stdout: index_type: 0 2024-05-06T13:28:05.848 INFO:journalctl@ceph.mon.b.smithi170.stdout: data_block_index_type: 0 2024-05-06T13:28:05.848 INFO:journalctl@ceph.mon.b.smithi170.stdout: index_shortening: 1 2024-05-06T13:28:05.848 INFO:journalctl@ceph.mon.b.smithi170.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-06T13:28:05.848 INFO:journalctl@ceph.mon.b.smithi170.stdout: hash_index_allow_collision: 1 2024-05-06T13:28:05.848 INFO:journalctl@ceph.mon.b.smithi170.stdout: checksum: 1 2024-05-06T13:28:05.848 INFO:journalctl@ceph.mon.b.smithi170.stdout: no_block_cache: 0 2024-05-06T13:28:05.848 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_cache: 0x55bc2dd32170 2024-05-06T13:28:05.848 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_cache_name: BinnedLRUCache 2024-05-06T13:28:05.848 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_cache_options: 2024-05-06T13:28:05.848 INFO:journalctl@ceph.mon.b.smithi170.stdout: capacity : 536870912 2024-05-06T13:28:05.848 INFO:journalctl@ceph.mon.b.smithi170.stdout: num_shard_bits : 4 2024-05-06T13:28:05.848 INFO:journalctl@ceph.mon.b.smithi170.stdout: strict_capacity_limit : 0 2024-05-06T13:28:05.848 INFO:journalctl@ceph.mon.b.smithi170.stdout: high_pri_pool_ratio: 0.000 2024-05-06T13:28:05.848 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_cache_compressed: (nil) 2024-05-06T13:28:05.848 INFO:journalctl@ceph.mon.b.smithi170.stdout: persistent_cache: (nil) 2024-05-06T13:28:05.848 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_size: 4096 2024-05-06T13:28:05.848 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_size_deviation: 10 2024-05-06T13:28:05.849 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_restart_interval: 16 2024-05-06T13:28:05.849 INFO:journalctl@ceph.mon.b.smithi170.stdout: index_block_restart_interval: 1 2024-05-06T13:28:05.849 INFO:journalctl@ceph.mon.b.smithi170.stdout: metadata_block_size: 4096 2024-05-06T13:28:05.849 INFO:journalctl@ceph.mon.b.smithi170.stdout: partition_filters: 0 2024-05-06T13:28:05.849 INFO:journalctl@ceph.mon.b.smithi170.stdout: use_delta_encoding: 1 2024-05-06T13:28:05.849 INFO:journalctl@ceph.mon.b.smithi170.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-05-06T13:28:05.849 INFO:journalctl@ceph.mon.b.smithi170.stdout: whole_key_filtering: 1 2024-05-06T13:28:05.849 INFO:journalctl@ceph.mon.b.smithi170.stdout: verify_compression: 0 2024-05-06T13:28:05.849 INFO:journalctl@ceph.mon.b.smithi170.stdout: read_amp_bytes_per_bit: 0 2024-05-06T13:28:05.849 INFO:journalctl@ceph.mon.b.smithi170.stdout: format_version: 4 2024-05-06T13:28:05.849 INFO:journalctl@ceph.mon.b.smithi170.stdout: enable_index_compression: 1 2024-05-06T13:28:05.849 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_align: 0 2024-05-06T13:28:05.849 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.write_buffer_size: 33554432 2024-05-06T13:28:05.849 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.max_write_buffer_number: 2 2024-05-06T13:28:05.849 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.compression: NoCompression 2024-05-06T13:28:05.850 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.bottommost_compression: Disabled 2024-05-06T13:28:05.850 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.prefix_extractor: nullptr 2024-05-06T13:28:05.850 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-06T13:28:05.850 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.num_levels: 7 2024-05-06T13:28:05.850 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-06T13:28:05.850 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-06T13:28:05.850 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-06T13:28:05.850 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-06T13:28:05.850 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-06T13:28:05.850 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-06T13:28:05.850 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-06T13:28:05.850 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-06T13:28:05.850 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-06T13:28:05.850 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-06T13:28:05.850 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-06T13:28:05.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.compression_opts.level: 32767 2024-05-06T13:28:05.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.compression_opts.strategy: 0 2024-05-06T13:28:05.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-06T13:28:05.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-06T13:28:05.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-06T13:28:05.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.compression_opts.enabled: false 2024-05-06T13:28:05.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-06T13:28:05.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-06T13:28:05.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-06T13:28:05.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.target_file_size_base: 67108864 2024-05-06T13:28:05.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-06T13:28:05.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-06T13:28:05.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-06T13:28:05.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-06T13:28:05.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-06T13:28:05.851 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-06T13:28:05.852 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-06T13:28:05.852 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-06T13:28:05.852 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-06T13:28:05.852 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-06T13:28:05.852 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-06T13:28:05.852 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-06T13:28:05.852 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-06T13:28:05.852 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.arena_block_size: 4194304 2024-05-06T13:28:05.852 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-06T13:28:05.852 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-06T13:28:05.852 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-05-06T13:28:05.852 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.disable_auto_compactions: 0 2024-05-06T13:28:05.852 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-06T13:28:05.852 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-06T13:28:05.852 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-06T13:28:05.852 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-06T13:28:05.852 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-06T13:28:05.853 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-06T13:28:05.853 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-06T13:28:05.853 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-06T13:28:05.853 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-06T13:28:05.853 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-06T13:28:05.853 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.table_properties_collectors: 2024-05-06T13:28:05.853 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.inplace_update_support: 0 2024-05-06T13:28:05.853 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-06T13:28:05.853 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-06T13:28:05.853 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-06T13:28:05.853 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-06T13:28:05.853 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.bloom_locality: 0 2024-05-06T13:28:05.853 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.max_successive_merges: 0 2024-05-06T13:28:05.853 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-06T13:28:05.853 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.paranoid_file_checks: 0 2024-05-06T13:28:05.853 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.force_consistency_checks: 1 2024-05-06T13:28:05.853 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.report_bg_io_stats: 0 2024-05-06T13:28:05.854 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.ttl: 2592000 2024-05-06T13:28:05.854 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-06T13:28:05.854 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.enable_blob_files: false 2024-05-06T13:28:05.854 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.min_blob_size: 0 2024-05-06T13:28:05.854 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.blob_file_size: 268435456 2024-05-06T13:28:05.854 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-06T13:28:05.854 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-06T13:28:05.854 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-06T13:28:05.854 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-06T13:28:05.854 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-05-06T13:28:05.854 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-05-06T13:28:05.854 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715002085820623, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-05-06T13:28:05.854 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-05-06T13:28:05.854 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-05-06T13:28:05.854 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715002085820928, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1670, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1715002085, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "7f08b7b2-1da9-4c9e-ad42-a93150ab2feb", "db_session_id": "SX7O0OYYAYK419P4E4ZY"}} 2024-05-06T13:28:05.854 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-05-06T13:28:05.855 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715002085821243, "job": 1, "event": "recovery_finished"} 2024-05-06T13:28:05.855 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-06T13:28:05.855 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55bc2dd18a80 2024-05-06T13:28:05.855 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: DB pointer 0x55bc2dd28000 2024-05-06T13:28:05.855 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-05-06T13:28:05.855 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-05-06T13:28:05.855 INFO:journalctl@ceph.mon.b.smithi170.stdout: ** DB Stats ** 2024-05-06T13:28:05.855 INFO:journalctl@ceph.mon.b.smithi170.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T13:28:05.855 INFO:journalctl@ceph.mon.b.smithi170.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-06T13:28:05.855 INFO:journalctl@ceph.mon.b.smithi170.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-06T13:28:05.855 INFO:journalctl@ceph.mon.b.smithi170.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-06T13:28:05.855 INFO:journalctl@ceph.mon.b.smithi170.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-06T13:28:05.855 INFO:journalctl@ceph.mon.b.smithi170.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-05-06T13:28:05.855 INFO:journalctl@ceph.mon.b.smithi170.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-06T13:28:05.855 INFO:journalctl@ceph.mon.b.smithi170.stdout: 2024-05-06T13:28:05.855 INFO:journalctl@ceph.mon.b.smithi170.stdout: ** Compaction Stats [default] ** 2024-05-06T13:28:05.856 INFO:journalctl@ceph.mon.b.smithi170.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-06T13:28:05.856 INFO:journalctl@ceph.mon.b.smithi170.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-06T13:28:05.856 INFO:journalctl@ceph.mon.b.smithi170.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-05-06T13:28:05.856 INFO:journalctl@ceph.mon.b.smithi170.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-05-06T13:28:05.856 INFO:journalctl@ceph.mon.b.smithi170.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-05-06T13:28:05.856 INFO:journalctl@ceph.mon.b.smithi170.stdout: 2024-05-06T13:28:05.856 INFO:journalctl@ceph.mon.b.smithi170.stdout: ** Compaction Stats [default] ** 2024-05-06T13:28:05.856 INFO:journalctl@ceph.mon.b.smithi170.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-06T13:28:05.856 INFO:journalctl@ceph.mon.b.smithi170.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-06T13:28:05.856 INFO:journalctl@ceph.mon.b.smithi170.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-05-06T13:28:05.856 INFO:journalctl@ceph.mon.b.smithi170.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T13:28:05.856 INFO:journalctl@ceph.mon.b.smithi170.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-06T13:28:05.856 INFO:journalctl@ceph.mon.b.smithi170.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-06T13:28:05.856 INFO:journalctl@ceph.mon.b.smithi170.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-06T13:28:05.856 INFO:journalctl@ceph.mon.b.smithi170.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-06T13:28:05.857 INFO:journalctl@ceph.mon.b.smithi170.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-06T13:28:05.857 INFO:journalctl@ceph.mon.b.smithi170.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T13:28:05.857 INFO:journalctl@ceph.mon.b.smithi170.stdout: Interval compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T13:28:05.857 INFO:journalctl@ceph.mon.b.smithi170.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-06T13:28:05.857 INFO:journalctl@ceph.mon.b.smithi170.stdout: 2024-05-06T13:28:05.857 INFO:journalctl@ceph.mon.b.smithi170.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-06T13:28:05.857 INFO:journalctl@ceph.mon.b.smithi170.stdout: 2024-05-06T13:28:05.857 INFO:journalctl@ceph.mon.b.smithi170.stdout: ** Compaction Stats [default] ** 2024-05-06T13:28:05.857 INFO:journalctl@ceph.mon.b.smithi170.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-06T13:28:05.857 INFO:journalctl@ceph.mon.b.smithi170.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-06T13:28:05.857 INFO:journalctl@ceph.mon.b.smithi170.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-05-06T13:28:05.857 INFO:journalctl@ceph.mon.b.smithi170.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-05-06T13:28:05.857 INFO:journalctl@ceph.mon.b.smithi170.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-05-06T13:28:05.857 INFO:journalctl@ceph.mon.b.smithi170.stdout: 2024-05-06T13:28:05.857 INFO:journalctl@ceph.mon.b.smithi170.stdout: ** Compaction Stats [default] ** 2024-05-06T13:28:05.857 INFO:journalctl@ceph.mon.b.smithi170.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-05-06T13:28:05.857 INFO:journalctl@ceph.mon.b.smithi170.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-06T13:28:05.858 INFO:journalctl@ceph.mon.b.smithi170.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.5 0.00 0.00 1 0.000 0 0 2024-05-06T13:28:05.858 INFO:journalctl@ceph.mon.b.smithi170.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T13:28:05.858 INFO:journalctl@ceph.mon.b.smithi170.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-06T13:28:05.858 INFO:journalctl@ceph.mon.b.smithi170.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-06T13:28:05.858 INFO:journalctl@ceph.mon.b.smithi170.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-06T13:28:05.858 INFO:journalctl@ceph.mon.b.smithi170.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-06T13:28:05.858 INFO:journalctl@ceph.mon.b.smithi170.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-06T13:28:05.858 INFO:journalctl@ceph.mon.b.smithi170.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T13:28:05.858 INFO:journalctl@ceph.mon.b.smithi170.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T13:28:05.858 INFO:journalctl@ceph.mon.b.smithi170.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-06T13:28:05.858 INFO:journalctl@ceph.mon.b.smithi170.stdout: 2024-05-06T13:28:05.858 INFO:journalctl@ceph.mon.b.smithi170.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-06T13:28:05.858 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-05-06T13:28:05.858 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: using public_addr v2:172.21.15.170:0/0 -> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2024-05-06T13:28:05.858 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: starting mon.b rank -1 at public addrs [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] at bind addrs [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:28:05.858 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: mon.b@-1(???) e0 preinit fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:28:05.859 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: mon.b@-1(synchronizing).mds e1 new map 2024-05-06T13:28:05.859 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: mon.b@-1(synchronizing).mds e1 print_map 2024-05-06T13:28:05.859 INFO:journalctl@ceph.mon.b.smithi170.stdout: e1 2024-05-06T13:28:05.859 INFO:journalctl@ceph.mon.b.smithi170.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-06T13:28:05.859 INFO:journalctl@ceph.mon.b.smithi170.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-06T13:28:05.859 INFO:journalctl@ceph.mon.b.smithi170.stdout: legacy client fscid: -1 2024-05-06T13:28:05.859 INFO:journalctl@ceph.mon.b.smithi170.stdout: 2024-05-06T13:28:05.859 INFO:journalctl@ceph.mon.b.smithi170.stdout: No filesystems configured 2024-05-06T13:28:05.859 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-06T13:28:05.859 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-06T13:28:05.859 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-06T13:28:05.859 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-06T13:28:05.859 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-06T13:28:05.859 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-06T13:28:05.859 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-06T13:28:05.859 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-06T13:28:05.860 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-06T13:28:05.860 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-06T13:28:05.860 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:05.860 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:28:05.860 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: mon.a calling monitor election 2024-05-06T13:28:05.860 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:28:05.860 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:28:05.860 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:05.860 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:28:05.860 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: mon.c calling monitor election 2024-05-06T13:28:05.860 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:28:05.860 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:05.860 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:28:05.860 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:28:05.860 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-06T13:28:05.861 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: monmap e2: 2 mons at {a=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0],c=[v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0]} 2024-05-06T13:28:05.861 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: fsmap 2024-05-06T13:28:05.861 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: osdmap e4: 0 total, 0 up, 0 in 2024-05-06T13:28:05.861 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: mgrmap e13: y(active, since 35s) 2024-05-06T13:28:05.861 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: overall HEALTH_OK 2024-05-06T13:28:05.861 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:05.861 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:28:05.861 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:05.861 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: Deploying daemon mon.b on smithi170 2024-05-06T13:28:05.861 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: from='client.? 172.21.15.170:0/864607528' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T13:28:05.861 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:28:05.861 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:05.861 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:05.861 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:05.861 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: from='client.? 172.21.15.170:0/905814282' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T13:28:05.862 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: mon.b@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-05-06T13:28:05.862 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 ceph-mon[23272]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-05-06T13:28:06.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 bash[23253]: 24be6fce9d28a1fec954db1eb8cf7b5583295d6e779e9ae383c08f400988d439 2024-05-06T13:28:06.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:05 smithi170 systemd[1]: Started Ceph mon.b for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:28:06.328 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-06T13:28:06.328 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-06T13:28:11.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:28:11.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[22428]: mon.a calling monitor election 2024-05-06T13:28:11.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[22428]: mon.c calling monitor election 2024-05-06T13:28:11.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:28:11.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:28:11.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[22428]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:11.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:28:11.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:28:11.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[22428]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:11.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:28:11.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:28:11.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:28:11.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[22428]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-06T13:28:11.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[22428]: monmap e3: 3 mons at {a=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0],b=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0],c=[v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0]} 2024-05-06T13:28:11.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[22428]: fsmap 2024-05-06T13:28:11.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[22428]: osdmap e4: 0 total, 0 up, 0 in 2024-05-06T13:28:11.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[22428]: mgrmap e13: y(active, since 46s) 2024-05-06T13:28:11.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[22428]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-05-06T13:28:11.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[22428]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-05-06T13:28:11.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[22428]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-05-06T13:28:11.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[22428]: mon.b (rank 2) addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] is down (out of quorum) 2024-05-06T13:28:11.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:11.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:28:11.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:11.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:28:11.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:28:11.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[27166]: mon.a calling monitor election 2024-05-06T13:28:11.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[27166]: mon.c calling monitor election 2024-05-06T13:28:11.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:28:11.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:28:11.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[27166]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:11.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:28:11.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:28:11.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[27166]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:11.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:28:11.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:28:11.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:28:11.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[27166]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-06T13:28:11.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[27166]: monmap e3: 3 mons at {a=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0],b=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0],c=[v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0]} 2024-05-06T13:28:11.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[27166]: fsmap 2024-05-06T13:28:11.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[27166]: osdmap e4: 0 total, 0 up, 0 in 2024-05-06T13:28:11.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[27166]: mgrmap e13: y(active, since 46s) 2024-05-06T13:28:11.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[27166]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-05-06T13:28:11.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[27166]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-05-06T13:28:11.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[27166]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-05-06T13:28:11.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[27166]: mon.b (rank 2) addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] is down (out of quorum) 2024-05-06T13:28:11.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:11.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:28:11.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:11.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:10 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:28:11.316 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-05-06T13:28:11.316 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":3,"fsid":"20e59cb4-0bac-11ef-bc97-c7b262605968","modified":"2024-05-06T13:28:05.841521Z","created":"2024-05-06T13:26:34.664804Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3301","nonce":0},{"type":"v1","addr":"172.21.15.77:6790","nonce":0}]},"addr":"172.21.15.77:6790/0","public_addr":"172.21.15.77:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-06T13:28:11.317 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 3 2024-05-06T13:28:11.959 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-05-06T13:28:11.960 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph config generate-minimal-conf 2024-05-06T13:28:12.257 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:11 smithi077 ceph-mon[22428]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:12.258 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:11 smithi077 ceph-mon[22428]: from='client.? 172.21.15.170:0/691107452' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T13:28:12.258 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:11 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:28:12.258 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:11 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:12.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:11 smithi077 ceph-mon[27166]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:12.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:11 smithi077 ceph-mon[27166]: from='client.? 172.21.15.170:0/691107452' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T13:28:12.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:11 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:28:12.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:11 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:13.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:28:13.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: mon.a calling monitor election 2024-05-06T13:28:13.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: mon.c calling monitor election 2024-05-06T13:28:13.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:28:13.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:28:13.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:13.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:28:13.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:28:13.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:13.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:28:13.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:28:13.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:28:13.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-06T13:28:13.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: monmap e3: 3 mons at {a=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0],b=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0],c=[v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0]} 2024-05-06T13:28:13.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: fsmap 2024-05-06T13:28:13.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: osdmap e4: 0 total, 0 up, 0 in 2024-05-06T13:28:13.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: mgrmap e13: y(active, since 46s) 2024-05-06T13:28:13.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-05-06T13:28:13.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-05-06T13:28:13.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-05-06T13:28:13.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: mon.b (rank 2) addr [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] is down (out of quorum) 2024-05-06T13:28:13.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:13.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:28:13.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:13.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:28:13.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:13.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: from='client.? 172.21.15.170:0/691107452' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-06T13:28:13.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:28:13.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:12 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:13.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[22428]: mon.b calling monitor election 2024-05-06T13:28:13.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[22428]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:13.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[22428]: Reconfiguring mon.a (unknown last config time)... 2024-05-06T13:28:13.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[22428]: Reconfiguring daemon mon.a on smithi077 2024-05-06T13:28:13.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[22428]: mon.b calling monitor election 2024-05-06T13:28:13.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[22428]: mon.c calling monitor election 2024-05-06T13:28:13.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[22428]: mon.a calling monitor election 2024-05-06T13:28:13.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[22428]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-06T13:28:13.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[22428]: monmap e3: 3 mons at {a=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0],b=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0],c=[v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0]} 2024-05-06T13:28:13.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[22428]: fsmap 2024-05-06T13:28:13.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[22428]: osdmap e4: 0 total, 0 up, 0 in 2024-05-06T13:28:13.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[22428]: mgrmap e13: y(active, since 48s) 2024-05-06T13:28:13.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[22428]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-05-06T13:28:13.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[22428]: Cluster is now healthy 2024-05-06T13:28:13.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[22428]: overall HEALTH_OK 2024-05-06T13:28:13.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[27166]: mon.b calling monitor election 2024-05-06T13:28:13.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[27166]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:13.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[27166]: Reconfiguring mon.a (unknown last config time)... 2024-05-06T13:28:13.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[27166]: Reconfiguring daemon mon.a on smithi077 2024-05-06T13:28:13.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[27166]: mon.b calling monitor election 2024-05-06T13:28:13.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[27166]: mon.c calling monitor election 2024-05-06T13:28:13.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[27166]: mon.a calling monitor election 2024-05-06T13:28:13.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[27166]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-06T13:28:13.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[27166]: monmap e3: 3 mons at {a=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0],b=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0],c=[v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0]} 2024-05-06T13:28:13.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[27166]: fsmap 2024-05-06T13:28:13.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[27166]: osdmap e4: 0 total, 0 up, 0 in 2024-05-06T13:28:13.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[27166]: mgrmap e13: y(active, since 48s) 2024-05-06T13:28:13.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[27166]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-05-06T13:28:13.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[27166]: Cluster is now healthy 2024-05-06T13:28:13.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:13 smithi077 ceph-mon[27166]: overall HEALTH_OK 2024-05-06T13:28:13.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:13 smithi170 ceph-mon[23272]: mon.b calling monitor election 2024-05-06T13:28:13.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:13 smithi170 ceph-mon[23272]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:13.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:13 smithi170 ceph-mon[23272]: Reconfiguring mon.a (unknown last config time)... 2024-05-06T13:28:13.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:13 smithi170 ceph-mon[23272]: Reconfiguring daemon mon.a on smithi077 2024-05-06T13:28:13.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:13 smithi170 ceph-mon[23272]: mon.b calling monitor election 2024-05-06T13:28:13.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:13 smithi170 ceph-mon[23272]: mon.c calling monitor election 2024-05-06T13:28:13.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:13 smithi170 ceph-mon[23272]: mon.a calling monitor election 2024-05-06T13:28:13.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:13 smithi170 ceph-mon[23272]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-06T13:28:13.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:13 smithi170 ceph-mon[23272]: monmap e3: 3 mons at {a=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0],b=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0],c=[v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0]} 2024-05-06T13:28:13.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:13 smithi170 ceph-mon[23272]: fsmap 2024-05-06T13:28:13.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:13 smithi170 ceph-mon[23272]: osdmap e4: 0 total, 0 up, 0 in 2024-05-06T13:28:13.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:13 smithi170 ceph-mon[23272]: mgrmap e13: y(active, since 48s) 2024-05-06T13:28:13.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:13 smithi170 ceph-mon[23272]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-05-06T13:28:13.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:13 smithi170 ceph-mon[23272]: Cluster is now healthy 2024-05-06T13:28:13.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:13 smithi170 ceph-mon[23272]: overall HEALTH_OK 2024-05-06T13:28:14.488 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:14 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:28:14.488 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:14 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:14.488 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:14 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:28:14.488 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:14 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:14.488 INFO:teuthology.orchestra.run.smithi077.stdout:# minimal ceph.conf for 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:28:14.489 INFO:teuthology.orchestra.run.smithi077.stdout:[global] 2024-05-06T13:28:14.489 INFO:teuthology.orchestra.run.smithi077.stdout: fsid = 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:28:14.489 INFO:teuthology.orchestra.run.smithi077.stdout: mon_host = [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] 2024-05-06T13:28:14.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:14 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:28:14.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:14 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:15.309 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-05-06T13:28:15.309 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-05-06T13:28:15.309 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.conf 2024-05-06T13:28:15.335 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-05-06T13:28:15.336 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:28:15.403 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-05-06T13:28:15.403 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.conf 2024-05-06T13:28:15.431 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-05-06T13:28:15.431 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:28:15.496 INFO:tasks.cephadm:Adding mgr.y on smithi077 2024-05-06T13:28:15.496 INFO:tasks.cephadm:Adding mgr.x on smithi170 2024-05-06T13:28:15.496 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph orch apply mgr '2;smithi077=y;smithi170=x' 2024-05-06T13:28:15.536 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:15 smithi077 ceph-mon[27166]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:15.536 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:15 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/4240327976' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:15.536 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:15 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:15.536 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:15 smithi077 ceph-mon[27166]: Reconfiguring mon.c (monmap changed)... 2024-05-06T13:28:15.537 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:15 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:28:15.537 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:15 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T13:28:15.537 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:15 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:15.537 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:15 smithi077 ceph-mon[27166]: Reconfiguring daemon mon.c on smithi077 2024-05-06T13:28:15.537 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:15 smithi077 ceph-mon[22428]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:15.537 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:15 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/4240327976' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:15.537 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:15 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:15.537 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:15 smithi077 ceph-mon[22428]: Reconfiguring mon.c (monmap changed)... 2024-05-06T13:28:15.537 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:15 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:28:15.537 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:15 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T13:28:15.537 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:15 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:15.537 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:15 smithi077 ceph-mon[22428]: Reconfiguring daemon mon.c on smithi077 2024-05-06T13:28:15.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:15 smithi170 ceph-mon[23272]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:15.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:15 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/4240327976' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:15.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:15 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:15.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:15 smithi170 ceph-mon[23272]: Reconfiguring mon.c (monmap changed)... 2024-05-06T13:28:15.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:15 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:28:15.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:15 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T13:28:15.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:15 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:15.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:15 smithi170 ceph-mon[23272]: Reconfiguring daemon mon.c on smithi077 2024-05-06T13:28:17.295 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled mgr update... 2024-05-06T13:28:17.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:17 smithi170 ceph-mon[23272]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:17.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:17 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:17.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:17 smithi170 ceph-mon[23272]: Reconfiguring mon.b (monmap changed)... 2024-05-06T13:28:17.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:17 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:28:17.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:17 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T13:28:17.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:17 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:17.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:17 smithi170 ceph-mon[23272]: Reconfiguring daemon mon.b on smithi170 2024-05-06T13:28:17.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:17 smithi170 ceph-mon[23272]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi077=y;smithi170=x", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:28:17.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:17 smithi170 ceph-mon[23272]: Saving service mgr spec with placement smithi077=y;smithi170=x;count:2 2024-05-06T13:28:17.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:17 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:17.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:17 smithi077 ceph-mon[22428]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:17.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:17 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:17.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:17 smithi077 ceph-mon[22428]: Reconfiguring mon.b (monmap changed)... 2024-05-06T13:28:17.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:17 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:28:17.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:17 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T13:28:17.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:17 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:17.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:17 smithi077 ceph-mon[22428]: Reconfiguring daemon mon.b on smithi170 2024-05-06T13:28:17.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:17 smithi077 ceph-mon[22428]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi077=y;smithi170=x", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:28:17.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:17 smithi077 ceph-mon[22428]: Saving service mgr spec with placement smithi077=y;smithi170=x;count:2 2024-05-06T13:28:17.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:17 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:17.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:17 smithi077 ceph-mon[27166]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:17.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:17 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:17.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:17 smithi077 ceph-mon[27166]: Reconfiguring mon.b (monmap changed)... 2024-05-06T13:28:17.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:17 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:28:17.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:17 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T13:28:17.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:17 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:17.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:17 smithi077 ceph-mon[27166]: Reconfiguring daemon mon.b on smithi170 2024-05-06T13:28:17.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:17 smithi077 ceph-mon[27166]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi077=y;smithi170=x", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:28:17.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:17 smithi077 ceph-mon[27166]: Saving service mgr spec with placement smithi077=y;smithi170=x;count:2 2024-05-06T13:28:17.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:17 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:18.313 DEBUG:teuthology.orchestra.run.smithi170:mgr.x> sudo journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mgr.x.service 2024-05-06T13:28:18.315 INFO:tasks.cephadm:Deploying OSDs... 2024-05-06T13:28:18.315 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-05-06T13:28:18.315 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2024-05-06T13:28:18.335 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-06T13:28:18.335 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_1 2024-05-06T13:28:18.393 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-06T13:28:18.393 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T13:28:18.393 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-05-06T13:28:18.393 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T13:28:18.393 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-05-06T13:28:18.393 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-05-06 13:27:54.472752148 +0000 2024-05-06T13:28:18.393 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-05-06 13:25:18.157295969 +0000 2024-05-06T13:28:18.393 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-05-06 13:25:18.157295969 +0000 2024-05-06T13:28:18.393 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-05-06 13:25:18.157295969 +0000 2024-05-06T13:28:18.394 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-06T13:28:18.461 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-05-06T13:28:18.461 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-05-06T13:28:18.462 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000238738 s, 2.1 MB/s 2024-05-06T13:28:18.463 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-06T13:28:18.521 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_2 2024-05-06T13:28:18.578 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-06T13:28:18.579 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T13:28:18.579 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-05-06T13:28:18.579 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T13:28:18.579 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-05-06T13:28:18.579 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-05-06 13:27:54.472752148 +0000 2024-05-06T13:28:18.579 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-05-06 13:25:18.519288431 +0000 2024-05-06T13:28:18.579 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-05-06 13:25:18.519288431 +0000 2024-05-06T13:28:18.579 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-05-06 13:25:18.519288431 +0000 2024-05-06T13:28:18.579 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-06T13:28:18.644 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-05-06T13:28:18.644 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-05-06T13:28:18.644 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.00023442 s, 2.2 MB/s 2024-05-06T13:28:18.646 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-06T13:28:18.704 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_3 2024-05-06T13:28:18.762 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-06T13:28:18.763 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T13:28:18.763 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-05-06T13:28:18.763 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T13:28:18.763 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-05-06T13:28:18.763 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-05-06 13:27:54.473752127 +0000 2024-05-06T13:28:18.763 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-05-06 13:25:18.853281475 +0000 2024-05-06T13:28:18.763 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-05-06 13:25:18.853281475 +0000 2024-05-06T13:28:18.763 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-05-06 13:25:18.853281475 +0000 2024-05-06T13:28:18.763 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-06T13:28:18.827 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-05-06T13:28:18.827 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-05-06T13:28:18.827 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000200765 s, 2.6 MB/s 2024-05-06T13:28:18.828 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-06T13:28:18.885 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_4 2024-05-06T13:28:18.941 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-06T13:28:18.941 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T13:28:18.941 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-05-06T13:28:18.941 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T13:28:18.942 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-05-06T13:28:18.942 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-05-06 13:27:54.473752127 +0000 2024-05-06T13:28:18.942 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-05-06 13:25:19.193274394 +0000 2024-05-06T13:28:18.942 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-05-06 13:25:19.193274394 +0000 2024-05-06T13:28:18.942 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-05-06 13:25:19.193274394 +0000 2024-05-06T13:28:18.942 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-06T13:28:19.005 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-05-06T13:28:19.005 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-05-06T13:28:19.005 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.00017515 s, 2.9 MB/s 2024-05-06T13:28:19.006 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-06T13:28:19.063 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-05-06T13:28:19.064 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/scratch_devs of=/dev/stdout 2024-05-06T13:28:19.079 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-06T13:28:19.079 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_1 2024-05-06T13:28:19.135 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-06T13:28:19.136 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T13:28:19.136 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-05-06T13:28:19.136 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T13:28:19.136 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-05-06T13:28:19.136 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-05-06 13:28:04.930150477 +0000 2024-05-06T13:28:19.136 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-05-06 13:25:18.151209767 +0000 2024-05-06T13:28:19.136 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-05-06 13:25:18.151209767 +0000 2024-05-06T13:28:19.136 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-05-06 13:25:18.151209767 +0000 2024-05-06T13:28:19.137 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-06T13:28:19.197 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-05-06T13:28:19.197 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-05-06T13:28:19.197 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 9.1375e-05 s, 5.6 MB/s 2024-05-06T13:28:19.199 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-06T13:28:19.254 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_2 2024-05-06T13:28:19.309 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-06T13:28:19.310 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T13:28:19.310 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-05-06T13:28:19.310 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T13:28:19.310 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-05-06T13:28:19.310 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-05-06 13:28:04.931150453 +0000 2024-05-06T13:28:19.310 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-05-06 13:25:18.489201549 +0000 2024-05-06T13:28:19.310 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-05-06 13:25:18.489201549 +0000 2024-05-06T13:28:19.310 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-05-06 13:25:18.489201549 +0000 2024-05-06T13:28:19.310 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-06T13:28:19.372 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-05-06T13:28:19.372 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-05-06T13:28:19.372 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000108062 s, 4.7 MB/s 2024-05-06T13:28:19.373 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-06T13:28:19.429 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_3 2024-05-06T13:28:19.484 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-06T13:28:19.484 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T13:28:19.484 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-05-06T13:28:19.484 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T13:28:19.484 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-05-06T13:28:19.485 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-05-06 13:28:04.931150453 +0000 2024-05-06T13:28:19.485 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-05-06 13:25:18.818193549 +0000 2024-05-06T13:28:19.485 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-05-06 13:25:18.818193549 +0000 2024-05-06T13:28:19.485 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-05-06 13:25:18.818193549 +0000 2024-05-06T13:28:19.485 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-06T13:28:19.547 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-05-06T13:28:19.547 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-05-06T13:28:19.547 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000107663 s, 4.8 MB/s 2024-05-06T13:28:19.548 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-06T13:28:19.603 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_4 2024-05-06T13:28:19.659 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-06T13:28:19.659 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-06T13:28:19.659 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-05-06T13:28:19.659 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-06T13:28:19.660 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-05-06T13:28:19.660 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-05-06 13:28:04.931150453 +0000 2024-05-06T13:28:19.660 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-05-06 13:25:19.152185429 +0000 2024-05-06T13:28:19.660 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-05-06 13:25:19.152185429 +0000 2024-05-06T13:28:19.660 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-05-06 13:25:19.152185429 +0000 2024-05-06T13:28:19.660 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-06T13:28:19.722 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-05-06T13:28:19.722 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-05-06T13:28:19.722 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.00010981 s, 4.7 MB/s 2024-05-06T13:28:19.723 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-06T13:28:19.779 INFO:tasks.cephadm:Deploying osd.0 on smithi077 with /dev/vg_nvme/lv_4... 2024-05-06T13:28:19.779 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-06T13:28:19.980 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:19 smithi170 ceph-mon[23272]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:19.980 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:19 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:19.980 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:19 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:28:19.980 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:19 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:19.980 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:19 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:28:19.980 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:19 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:19.980 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:19 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T13:28:19.980 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:19 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-06T13:28:19.981 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:19 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T13:28:19.981 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:19 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:19.981 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:19 smithi170 ceph-mon[23272]: Deploying daemon mgr.x on smithi170 2024-05-06T13:28:20.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:19 smithi077 ceph-mon[22428]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:20.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:19 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:20.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:19 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:28:20.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:19 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:20.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:19 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:28:20.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:19 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:20.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:19 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T13:28:20.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:19 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-06T13:28:20.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:19 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T13:28:20.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:19 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:20.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:19 smithi077 ceph-mon[22428]: Deploying daemon mgr.x on smithi170 2024-05-06T13:28:20.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:19 smithi077 ceph-mon[27166]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:20.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:19 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:20.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:19 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:28:20.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:19 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:20.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:19 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:28:20.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:19 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:20.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:19 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T13:28:20.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:19 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-06T13:28:20.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:19 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T13:28:20.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:19 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:20.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:19 smithi077 ceph-mon[27166]: Deploying daemon mgr.x on smithi170 2024-05-06T13:28:20.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:20 smithi170 podman[24434]: 2024-05-06 13:28:20.518649267 +0000 UTC m=+0.117657492 container create 82c841abfafd7ae48579ff72a3c2353a22dda83f9e02ba8dccb1bbd5b7ab983b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x, ceph=True, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, GIT_BRANCH=HEAD, vcs-type=git, version=8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container) 2024-05-06T13:28:20.865 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:20 smithi170 podman[24434]: 2024-05-06 13:28:20.721214778 +0000 UTC m=+0.320223012 container init 82c841abfafd7ae48579ff72a3c2353a22dda83f9e02ba8dccb1bbd5b7ab983b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x, RELEASE=HEAD, name=centos-stream, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., release=754, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, architecture=x86_64, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, ceph=True) 2024-05-06T13:28:20.865 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:20 smithi170 podman[24434]: 2024-05-06 13:28:20.724152901 +0000 UTC m=+0.323161135 container start 82c841abfafd7ae48579ff72a3c2353a22dda83f9e02ba8dccb1bbd5b7ab983b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x, ceph=True, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, io.buildah.version=1.19.8, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, vcs-type=git, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, RELEASE=HEAD) 2024-05-06T13:28:20.865 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:20 smithi170 ceph-mgr[24453]: -- 172.21.15.170:0/3243121002 <== mon.2 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555a56b86340 con 0x555a56b5e800 2024-05-06T13:28:20.865 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:20 smithi170 bash[24434]: 82c841abfafd7ae48579ff72a3c2353a22dda83f9e02ba8dccb1bbd5b7ab983b 2024-05-06T13:28:20.865 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:20 smithi170 systemd[1]: Started Ceph mgr.x for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:28:20.865 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:28:20.864+0000 7f944698f000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T13:28:21.325 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:28:20.998+0000 7f944698f000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:28:21.350 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:21 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:28:21.099+0000 7f944698f000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T13:28:21.774 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-06T13:28:21.774 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-06T13:28:21.774 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-05-06T13:28:21.774 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-05-06T13:28:21.774 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125116 s, 838 MB/s 2024-05-06T13:28:21.774 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-05-06T13:28:21.777 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:21 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:28:21.428+0000 7f944698f000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T13:28:21.777 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:21 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:28:21.514+0000 7f944698f000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T13:28:22.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:21 smithi077 ceph-mon[22428]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:22.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:21 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:22.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:21 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:28:22.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:21 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:22.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:21 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:28:22.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:21 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:22.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:21 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:22.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:21 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T13:28:22.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:21 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T13:28:22.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:21 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:22.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:21 smithi077 ceph-mon[27166]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:22.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:21 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:22.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:21 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:28:22.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:21 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:22.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:21 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:28:22.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:21 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:22.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:21 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:22.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:21 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T13:28:22.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:21 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T13:28:22.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:21 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:22.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:21 smithi170 ceph-mon[23272]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:22.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:21 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:22.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:21 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:28:22.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:21 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:22.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:21 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:28:22.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:21 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:22.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:21 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:22.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:21 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T13:28:22.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:21 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T13:28:22.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:21 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:22.460 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi077:vg_nvme/lv_4 2024-05-06T13:28:23.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:22 smithi077 ceph-mon[22428]: Reconfiguring mgr.y (unknown last config time)... 2024-05-06T13:28:23.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:22 smithi077 ceph-mon[22428]: Reconfiguring daemon mgr.y on smithi077 2024-05-06T13:28:23.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:22 smithi077 ceph-mon[27166]: Reconfiguring mgr.y (unknown last config time)... 2024-05-06T13:28:23.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:22 smithi077 ceph-mon[27166]: Reconfiguring daemon mgr.y on smithi077 2024-05-06T13:28:23.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:22 smithi170 ceph-mon[23272]: Reconfiguring mgr.y (unknown last config time)... 2024-05-06T13:28:23.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:22 smithi170 ceph-mon[23272]: Reconfiguring daemon mgr.y on smithi077 2024-05-06T13:28:23.076 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:22 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:28:22.659+0000 7f944698f000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T13:28:23.076 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:22 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:28:22.820+0000 7f944698f000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T13:28:23.076 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:22 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:28:22.902+0000 7f944698f000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T13:28:23.575 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:23 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:28:23.213+0000 7f944698f000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T13:28:23.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:23 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:28:23.303+0000 7f944698f000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T13:28:23.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:23 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:28:23.394+0000 7f944698f000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T13:28:23.887 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:23 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:28:23.634+0000 7f944698f000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:28:24.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:23 smithi077 ceph-mon[22428]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:24.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:23 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:24.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:23 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:28:24.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:23 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:24.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:23 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:28:24.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:23 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:24.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:23 smithi077 ceph-mon[27166]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:24.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:23 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:24.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:23 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:28:24.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:23 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:24.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:23 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:28:24.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:23 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:24.325 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:28:24.072+0000 7f944698f000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T13:28:24.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:23 smithi170 ceph-mon[23272]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:24.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:23 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:24.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:23 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:28:24.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:23 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:24.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:23 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:28:24.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:23 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:24.825 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:28:24.556+0000 7f944698f000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T13:28:24.825 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:28:24.648+0000 7f944698f000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T13:28:25.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:24 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:28:25.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:24 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:28:25.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:24 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:25.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:24 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:28:25.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:24 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:28:25.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:24 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:25.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:24 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:28:25.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:24 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:28:25.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:24 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:25.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:28:24.938+0000 7f944698f000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T13:28:25.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:28:25.025+0000 7f944698f000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T13:28:25.778 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:28:25.449+0000 7f944698f000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T13:28:25.778 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:28:25.579+0000 7f944698f000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T13:28:25.779 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:28:25.667+0000 7f944698f000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T13:28:25.779 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:28:25.778+0000 7f944698f000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T13:28:26.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:25 smithi077 ceph-mon[27166]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:26.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:25 smithi077 ceph-mon[27166]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:28:26.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:25 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T13:28:26.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:25 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T13:28:26.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:25 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:26.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:25 smithi077 ceph-mon[22428]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:26.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:25 smithi077 ceph-mon[22428]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:28:26.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:25 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T13:28:26.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:25 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T13:28:26.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:25 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:26.075 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:28:25.865+0000 7f944698f000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T13:28:26.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:25 smithi170 ceph-mon[23272]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:26.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:25 smithi170 ceph-mon[23272]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:28:26.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:25 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T13:28:26.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:25 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T13:28:26.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:25 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:26.764 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:26 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:28:26.401+0000 7f944698f000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T13:28:27.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:26 smithi077 ceph-mon[22428]: Standby manager daemon x started 2024-05-06T13:28:27.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:26 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.170:0/3945824188' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T13:28:27.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:26 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.170:0/3945824188' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:28:27.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:26 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.170:0/3945824188' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T13:28:27.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:26 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.170:0/3945824188' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:28:27.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:26 smithi077 ceph-mon[27166]: Standby manager daemon x started 2024-05-06T13:28:27.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:26 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/3945824188' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T13:28:27.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:26 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/3945824188' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:28:27.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:26 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/3945824188' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T13:28:27.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:26 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/3945824188' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:28:27.075 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:28:26 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:28:26.764+0000 7f944698f000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T13:28:27.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:26 smithi170 ceph-mon[23272]: Standby manager daemon x started 2024-05-06T13:28:27.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:26 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.170:0/3945824188' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T13:28:27.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:26 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.170:0/3945824188' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:28:27.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:26 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.170:0/3945824188' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T13:28:27.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:26 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.170:0/3945824188' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:28:28.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:27 smithi077 ceph-mon[22428]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:28.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:27 smithi077 ceph-mon[22428]: mgrmap e14: y(active, since 62s), standbys: x 2024-05-06T13:28:28.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:27 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T13:28:28.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:27 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/593306163' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8baf11f2-252b-4937-8684-28c621b69971"}]: dispatch 2024-05-06T13:28:28.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:27 smithi077 ceph-mon[22428]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8baf11f2-252b-4937-8684-28c621b69971"}]: dispatch 2024-05-06T13:28:28.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:27 smithi077 ceph-mon[22428]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8baf11f2-252b-4937-8684-28c621b69971"}]': finished 2024-05-06T13:28:28.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:27 smithi077 ceph-mon[22428]: osdmap e5: 1 total, 0 up, 1 in 2024-05-06T13:28:28.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:27 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:28:28.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:27 smithi077 ceph-mon[27166]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:28.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:27 smithi077 ceph-mon[27166]: mgrmap e14: y(active, since 62s), standbys: x 2024-05-06T13:28:28.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:27 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T13:28:28.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:27 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/593306163' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8baf11f2-252b-4937-8684-28c621b69971"}]: dispatch 2024-05-06T13:28:28.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:27 smithi077 ceph-mon[27166]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8baf11f2-252b-4937-8684-28c621b69971"}]: dispatch 2024-05-06T13:28:28.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:27 smithi077 ceph-mon[27166]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8baf11f2-252b-4937-8684-28c621b69971"}]': finished 2024-05-06T13:28:28.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:27 smithi077 ceph-mon[27166]: osdmap e5: 1 total, 0 up, 1 in 2024-05-06T13:28:28.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:27 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:28:28.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:27 smithi170 ceph-mon[23272]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:28.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:27 smithi170 ceph-mon[23272]: mgrmap e14: y(active, since 62s), standbys: x 2024-05-06T13:28:28.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:27 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T13:28:28.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:27 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/593306163' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8baf11f2-252b-4937-8684-28c621b69971"}]: dispatch 2024-05-06T13:28:28.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:27 smithi170 ceph-mon[23272]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8baf11f2-252b-4937-8684-28c621b69971"}]: dispatch 2024-05-06T13:28:28.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:27 smithi170 ceph-mon[23272]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8baf11f2-252b-4937-8684-28c621b69971"}]': finished 2024-05-06T13:28:28.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:27 smithi170 ceph-mon[23272]: osdmap e5: 1 total, 0 up, 1 in 2024-05-06T13:28:28.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:27 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:28:29.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:28 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/690786690' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T13:28:29.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:28 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/690786690' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T13:28:29.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:28 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/690786690' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T13:28:30.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:29 smithi077 ceph-mon[22428]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:30.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:29 smithi077 ceph-mon[27166]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:30.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:29 smithi170 ceph-mon[23272]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:32.042 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:31 smithi077 ceph-mon[22428]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:32.042 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:31 smithi077 ceph-mon[27166]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:32.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:31 smithi170 ceph-mon[23272]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:34.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:33 smithi077 ceph-mon[22428]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:34.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:33 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-06T13:28:34.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:33 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:34.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:33 smithi077 ceph-mon[22428]: Deploying daemon osd.0 on smithi077 2024-05-06T13:28:34.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:33 smithi077 ceph-mon[27166]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:34.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:33 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-06T13:28:34.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:33 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:34.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:33 smithi077 ceph-mon[27166]: Deploying daemon osd.0 on smithi077 2024-05-06T13:28:34.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:33 smithi170 ceph-mon[23272]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:34.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:33 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-06T13:28:34.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:33 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:34.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:33 smithi170 ceph-mon[23272]: Deploying daemon osd.0 on smithi077 2024-05-06T13:28:35.957 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:35 smithi077 ceph-mon[22428]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:35.957 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:35 smithi077 ceph-mon[27166]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:36.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:35 smithi170 ceph-mon[23272]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:38.198 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:37 smithi077 ceph-mon[22428]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:38.198 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:37 smithi077 ceph-mon[27166]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:38.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:37 smithi170 ceph-mon[23272]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:39.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:39 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:39.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:39 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:28:39.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:39 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:39.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:39 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:28:39.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:39 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:39.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:39 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:28:39.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:39 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:39.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:39 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:28:39.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:39 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:39.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:39 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:28:39.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:39 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:39.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:39 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:28:40.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:40 smithi077 ceph-mon[22428]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:40.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:40 smithi077 ceph-mon[27166]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:40.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:40 smithi170 ceph-mon[23272]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:41.601 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 0 on host 'smithi077' 2024-05-06T13:28:41.899 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:41 smithi077 ceph-mon[27166]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:41.900 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:41 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:41.900 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:41 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:41.900 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:41 smithi077 ceph-mon[27166]: from='osd.0 [v2:172.21.15.77:6802/834793539,v1:172.21.15.77:6803/834793539]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-06T13:28:41.900 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:41 smithi077 ceph-mon[27166]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-06T13:28:41.900 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:41 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:41.900 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:41 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:28:41.900 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:41 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:41.900 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:41 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:28:41.900 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:41 smithi077 ceph-mon[22428]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:41.901 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:41 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:41.901 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:41 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:41.901 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:41 smithi077 ceph-mon[22428]: from='osd.0 [v2:172.21.15.77:6802/834793539,v1:172.21.15.77:6803/834793539]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-06T13:28:41.901 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:41 smithi077 ceph-mon[22428]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-06T13:28:41.901 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:41 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:41.901 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:41 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:28:41.901 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:41 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:41.901 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:41 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:28:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:41 smithi170 ceph-mon[23272]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:41 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:41 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:41 smithi170 ceph-mon[23272]: from='osd.0 [v2:172.21.15.77:6802/834793539,v1:172.21.15.77:6803/834793539]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-06T13:28:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:41 smithi170 ceph-mon[23272]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-06T13:28:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:41 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:41 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:28:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:41 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:41 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:28:42.187 DEBUG:teuthology.orchestra.run.smithi077:osd.0> sudo journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.0.service 2024-05-06T13:28:42.189 INFO:tasks.cephadm:Deploying osd.1 on smithi077 with /dev/vg_nvme/lv_3... 2024-05-06T13:28:42.189 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-06T13:28:43.293 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:43 smithi077 ceph-mon[22428]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-06T13:28:43.293 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:43 smithi077 ceph-mon[22428]: osdmap e6: 1 total, 0 up, 1 in 2024-05-06T13:28:43.293 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:43 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:28:43.293 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:43 smithi077 ceph-mon[22428]: from='osd.0 [v2:172.21.15.77:6802/834793539,v1:172.21.15.77:6803/834793539]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T13:28:43.293 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:43 smithi077 ceph-mon[22428]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T13:28:43.294 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:43 smithi077 ceph-mon[27166]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-06T13:28:43.294 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:43 smithi077 ceph-mon[27166]: osdmap e6: 1 total, 0 up, 1 in 2024-05-06T13:28:43.294 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:43 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:28:43.294 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:43 smithi077 ceph-mon[27166]: from='osd.0 [v2:172.21.15.77:6802/834793539,v1:172.21.15.77:6803/834793539]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T13:28:43.294 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:43 smithi077 ceph-mon[27166]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T13:28:43.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:28:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0[30015]: 2024-05-06T13:28:43.197+0000 7f3f6abb1700 -1 osd.0 0 waiting for initial osdmap 2024-05-06T13:28:43.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:28:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0[30015]: 2024-05-06T13:28:43.200+0000 7f3f65d49700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T13:28:43.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:43 smithi170 ceph-mon[23272]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-06T13:28:43.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:43 smithi170 ceph-mon[23272]: osdmap e6: 1 total, 0 up, 1 in 2024-05-06T13:28:43.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:43 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:28:43.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:43 smithi170 ceph-mon[23272]: from='osd.0 [v2:172.21.15.77:6802/834793539,v1:172.21.15.77:6803/834793539]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T13:28:43.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:43 smithi170 ceph-mon[23272]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T13:28:44.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:44 smithi077 ceph-mon[22428]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:44.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:44 smithi077 ceph-mon[22428]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-05-06T13:28:44.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:44 smithi077 ceph-mon[22428]: osdmap e7: 1 total, 0 up, 1 in 2024-05-06T13:28:44.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:44 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:28:44.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:44 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:28:44.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:44 smithi077 ceph-mon[27166]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:44.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:44 smithi077 ceph-mon[27166]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-05-06T13:28:44.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:44 smithi077 ceph-mon[27166]: osdmap e7: 1 total, 0 up, 1 in 2024-05-06T13:28:44.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:44 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:28:44.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:44 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:28:44.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:44 smithi170 ceph-mon[23272]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:44.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:44 smithi170 ceph-mon[23272]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-05-06T13:28:44.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:44 smithi170 ceph-mon[23272]: osdmap e7: 1 total, 0 up, 1 in 2024-05-06T13:28:44.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:44 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:28:44.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:44 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:28:44.594 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-06T13:28:44.594 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-06T13:28:44.594 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-05-06T13:28:44.594 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-05-06T13:28:44.594 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123117 s, 852 MB/s 2024-05-06T13:28:44.594 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-05-06T13:28:45.281 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:45 smithi077 ceph-mon[22428]: purged_snaps scrub starts 2024-05-06T13:28:45.281 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:45 smithi077 ceph-mon[22428]: purged_snaps scrub ok 2024-05-06T13:28:45.281 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:45 smithi077 ceph-mon[22428]: osd.0 [v2:172.21.15.77:6802/834793539,v1:172.21.15.77:6803/834793539] boot 2024-05-06T13:28:45.281 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:45 smithi077 ceph-mon[22428]: osdmap e8: 1 total, 1 up, 1 in 2024-05-06T13:28:45.281 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:45 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:28:45.281 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:45 smithi077 ceph-mon[27166]: purged_snaps scrub starts 2024-05-06T13:28:45.281 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:45 smithi077 ceph-mon[27166]: purged_snaps scrub ok 2024-05-06T13:28:45.281 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:45 smithi077 ceph-mon[27166]: osd.0 [v2:172.21.15.77:6802/834793539,v1:172.21.15.77:6803/834793539] boot 2024-05-06T13:28:45.281 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:45 smithi077 ceph-mon[27166]: osdmap e8: 1 total, 1 up, 1 in 2024-05-06T13:28:45.282 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:45 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:28:45.309 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi077:vg_nvme/lv_3 2024-05-06T13:28:45.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:45 smithi170 ceph-mon[23272]: purged_snaps scrub starts 2024-05-06T13:28:45.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:45 smithi170 ceph-mon[23272]: purged_snaps scrub ok 2024-05-06T13:28:45.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:45 smithi170 ceph-mon[23272]: osd.0 [v2:172.21.15.77:6802/834793539,v1:172.21.15.77:6803/834793539] boot 2024-05-06T13:28:45.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:45 smithi170 ceph-mon[23272]: osdmap e8: 1 total, 1 up, 1 in 2024-05-06T13:28:45.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:45 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:28:46.360 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:46 smithi077 ceph-mon[22428]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:46.360 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:46 smithi077 ceph-mon[22428]: osdmap e9: 1 total, 1 up, 1 in 2024-05-06T13:28:46.360 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:46 smithi077 ceph-mon[27166]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:46.360 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:46 smithi077 ceph-mon[27166]: osdmap e9: 1 total, 1 up, 1 in 2024-05-06T13:28:46.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:46 smithi170 ceph-mon[23272]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-06T13:28:46.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:46 smithi170 ceph-mon[23272]: osdmap e9: 1 total, 1 up, 1 in 2024-05-06T13:28:48.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:47 smithi077 ceph-mon[22428]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:28:48.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:47 smithi077 ceph-mon[22428]: Detected new or changed devices on smithi077 2024-05-06T13:28:48.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:47 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:48.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:47 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:28:48.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:47 smithi077 ceph-mon[22428]: Adjusting osd_memory_target on smithi077 to 16064M 2024-05-06T13:28:48.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:47 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:48.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:47 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:48.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:47 smithi077 ceph-mon[27166]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:28:48.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:47 smithi077 ceph-mon[27166]: Detected new or changed devices on smithi077 2024-05-06T13:28:48.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:47 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:48.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:47 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:28:48.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:47 smithi077 ceph-mon[27166]: Adjusting osd_memory_target on smithi077 to 16064M 2024-05-06T13:28:48.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:47 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:48.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:47 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:48.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:47 smithi170 ceph-mon[23272]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:28:48.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:47 smithi170 ceph-mon[23272]: Detected new or changed devices on smithi077 2024-05-06T13:28:48.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:47 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:48.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:47 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:28:48.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:47 smithi170 ceph-mon[23272]: Adjusting osd_memory_target on smithi077 to 16064M 2024-05-06T13:28:48.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:47 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:48.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:47 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:28:49.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:49 smithi077 ceph-mon[27166]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:28:49.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:49 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T13:28:49.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:49 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T13:28:49.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:49 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:49.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:49 smithi077 ceph-mon[22428]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:28:49.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:49 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T13:28:49.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:49 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T13:28:49.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:49 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:49.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:49 smithi170 ceph-mon[23272]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:28:49.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:49 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T13:28:49.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:49 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T13:28:49.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:49 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:50.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:50 smithi077 ceph-mon[22428]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:28:50.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:50 smithi077 ceph-mon[27166]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:28:50.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:50 smithi170 ceph-mon[23272]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:28:51.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:51 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/309810585' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "69b4ced2-2786-41d0-be2d-2acfcb52ece1"}]: dispatch 2024-05-06T13:28:51.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:51 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/309810585' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "69b4ced2-2786-41d0-be2d-2acfcb52ece1"}]': finished 2024-05-06T13:28:51.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:51 smithi077 ceph-mon[22428]: osdmap e10: 2 total, 1 up, 2 in 2024-05-06T13:28:51.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:51 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:28:51.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:51 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3911968377' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T13:28:51.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:51 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/309810585' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "69b4ced2-2786-41d0-be2d-2acfcb52ece1"}]: dispatch 2024-05-06T13:28:51.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:51 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/309810585' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "69b4ced2-2786-41d0-be2d-2acfcb52ece1"}]': finished 2024-05-06T13:28:51.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:51 smithi077 ceph-mon[27166]: osdmap e10: 2 total, 1 up, 2 in 2024-05-06T13:28:51.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:51 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:28:51.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:51 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3911968377' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T13:28:51.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:51 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/309810585' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "69b4ced2-2786-41d0-be2d-2acfcb52ece1"}]: dispatch 2024-05-06T13:28:51.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:51 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/309810585' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "69b4ced2-2786-41d0-be2d-2acfcb52ece1"}]': finished 2024-05-06T13:28:51.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:51 smithi170 ceph-mon[23272]: osdmap e10: 2 total, 1 up, 2 in 2024-05-06T13:28:51.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:51 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:28:51.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:51 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/3911968377' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T13:28:52.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:52 smithi077 ceph-mon[22428]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:28:52.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:52 smithi077 ceph-mon[27166]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:28:52.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:52 smithi170 ceph-mon[23272]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:28:54.297 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:54 smithi077 ceph-mon[22428]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:28:54.297 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:54 smithi077 ceph-mon[27166]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:28:54.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:54 smithi170 ceph-mon[23272]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:28:56.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:56 smithi077 ceph-mon[22428]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:28:56.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:56 smithi077 ceph-mon[27166]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:28:56.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:56 smithi170 ceph-mon[23272]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:28:57.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:57 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-06T13:28:57.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:57 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:57.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:57 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-06T13:28:57.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:57 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:57.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:57 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-06T13:28:57.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:57 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:28:58.212 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:58 smithi077 ceph-mon[22428]: Deploying daemon osd.1 on smithi077 2024-05-06T13:28:58.212 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:28:58 smithi077 ceph-mon[22428]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:28:58.213 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:58 smithi077 ceph-mon[27166]: Deploying daemon osd.1 on smithi077 2024-05-06T13:28:58.213 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:28:58 smithi077 ceph-mon[27166]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:28:58.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:58 smithi170 ceph-mon[23272]: Deploying daemon osd.1 on smithi077 2024-05-06T13:28:58.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:28:58 smithi170 ceph-mon[23272]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:29:00.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:00 smithi077 ceph-mon[22428]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:29:00.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:00 smithi077 ceph-mon[27166]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:29:00.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:00 smithi170 ceph-mon[23272]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:29:02.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:02 smithi077 ceph-mon[22428]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:29:02.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:02 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:02.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:02 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:29:02.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:02 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:02.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:02 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:29:02.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:02 smithi077 ceph-mon[27166]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:29:02.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:02 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:02.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:02 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:29:02.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:02 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:02.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:02 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:29:02.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:02 smithi170 ceph-mon[23272]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:29:02.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:02 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:02.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:02 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:29:02.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:02 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:02.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:02 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:29:04.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:04 smithi077 ceph-mon[22428]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:29:04.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:04 smithi077 ceph-mon[27166]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:29:04.336 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 1 on host 'smithi077' 2024-05-06T13:29:04.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:04 smithi170 ceph-mon[23272]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:29:04.911 DEBUG:teuthology.orchestra.run.smithi077:osd.1> sudo journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.1.service 2024-05-06T13:29:04.913 INFO:tasks.cephadm:Deploying osd.2 on smithi077 with /dev/vg_nvme/lv_2... 2024-05-06T13:29:04.913 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-06T13:29:05.545 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:05 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:05.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:05 smithi077 ceph-mon[22428]: from='osd.1 [v2:172.21.15.77:6810/2133267326,v1:172.21.15.77:6811/2133267326]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-06T13:29:05.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:05 smithi077 ceph-mon[22428]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:29:05.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:05 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:05.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:05 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:05.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:05 smithi077 ceph-mon[27166]: from='osd.1 [v2:172.21.15.77:6810/2133267326,v1:172.21.15.77:6811/2133267326]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-06T13:29:05.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:05 smithi077 ceph-mon[27166]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:29:05.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:05 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:05.625 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:05 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:05.625 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:05 smithi170 ceph-mon[23272]: from='osd.1 [v2:172.21.15.77:6810/2133267326,v1:172.21.15.77:6811/2133267326]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-06T13:29:05.625 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:05 smithi170 ceph-mon[23272]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:29:05.625 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:05 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:06.644 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:06 smithi077 ceph-mon[22428]: from='osd.1 [v2:172.21.15.77:6810/2133267326,v1:172.21.15.77:6811/2133267326]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-06T13:29:06.644 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:06 smithi077 ceph-mon[22428]: osdmap e11: 2 total, 1 up, 2 in 2024-05-06T13:29:06.644 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:06 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:29:06.644 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:06 smithi077 ceph-mon[22428]: from='osd.1 [v2:172.21.15.77:6810/2133267326,v1:172.21.15.77:6811/2133267326]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T13:29:06.645 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:06 smithi077 ceph-mon[27166]: from='osd.1 [v2:172.21.15.77:6810/2133267326,v1:172.21.15.77:6811/2133267326]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-06T13:29:06.645 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:06 smithi077 ceph-mon[27166]: osdmap e11: 2 total, 1 up, 2 in 2024-05-06T13:29:06.645 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:06 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:29:06.645 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:06 smithi077 ceph-mon[27166]: from='osd.1 [v2:172.21.15.77:6810/2133267326,v1:172.21.15.77:6811/2133267326]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T13:29:06.645 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 13:29:06 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1[32976]: 2024-05-06T13:29:06.345+0000 7fd504bac700 -1 osd.1 0 waiting for initial osdmap 2024-05-06T13:29:06.645 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 13:29:06 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1[32976]: 2024-05-06T13:29:06.349+0000 7fd4fdd40700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T13:29:06.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:06 smithi170 ceph-mon[23272]: from='osd.1 [v2:172.21.15.77:6810/2133267326,v1:172.21.15.77:6811/2133267326]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-06T13:29:06.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:06 smithi170 ceph-mon[23272]: osdmap e11: 2 total, 1 up, 2 in 2024-05-06T13:29:06.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:06 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:29:06.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:06 smithi170 ceph-mon[23272]: from='osd.1 [v2:172.21.15.77:6810/2133267326,v1:172.21.15.77:6811/2133267326]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T13:29:07.637 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:07 smithi077 ceph-mon[22428]: purged_snaps scrub starts 2024-05-06T13:29:07.637 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:07 smithi077 ceph-mon[22428]: purged_snaps scrub ok 2024-05-06T13:29:07.637 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:07 smithi077 ceph-mon[22428]: from='osd.1 [v2:172.21.15.77:6810/2133267326,v1:172.21.15.77:6811/2133267326]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-05-06T13:29:07.638 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:07 smithi077 ceph-mon[22428]: osdmap e12: 2 total, 1 up, 2 in 2024-05-06T13:29:07.638 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:07 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:29:07.638 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:07 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:29:07.638 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:07 smithi077 ceph-mon[22428]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:29:07.638 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:07 smithi077 ceph-mon[27166]: purged_snaps scrub starts 2024-05-06T13:29:07.638 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:07 smithi077 ceph-mon[27166]: purged_snaps scrub ok 2024-05-06T13:29:07.638 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:07 smithi077 ceph-mon[27166]: from='osd.1 [v2:172.21.15.77:6810/2133267326,v1:172.21.15.77:6811/2133267326]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-05-06T13:29:07.638 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:07 smithi077 ceph-mon[27166]: osdmap e12: 2 total, 1 up, 2 in 2024-05-06T13:29:07.638 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:07 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:29:07.638 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:07 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:29:07.638 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:07 smithi077 ceph-mon[27166]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:29:07.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:07 smithi170 ceph-mon[23272]: purged_snaps scrub starts 2024-05-06T13:29:07.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:07 smithi170 ceph-mon[23272]: purged_snaps scrub ok 2024-05-06T13:29:07.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:07 smithi170 ceph-mon[23272]: from='osd.1 [v2:172.21.15.77:6810/2133267326,v1:172.21.15.77:6811/2133267326]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-05-06T13:29:07.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:07 smithi170 ceph-mon[23272]: osdmap e12: 2 total, 1 up, 2 in 2024-05-06T13:29:07.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:07 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:29:07.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:07 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:29:07.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:07 smithi170 ceph-mon[23272]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-05-06T13:29:08.396 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-06T13:29:08.396 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-06T13:29:08.396 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-05-06T13:29:08.396 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-05-06T13:29:08.396 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0151007 s, 694 MB/s 2024-05-06T13:29:08.397 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-05-06T13:29:08.715 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:08 smithi077 ceph-mon[27166]: osd.1 [v2:172.21.15.77:6810/2133267326,v1:172.21.15.77:6811/2133267326] boot 2024-05-06T13:29:08.715 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:08 smithi077 ceph-mon[27166]: osdmap e13: 2 total, 2 up, 2 in 2024-05-06T13:29:08.716 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:08 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:29:08.716 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:08 smithi077 ceph-mon[22428]: osd.1 [v2:172.21.15.77:6810/2133267326,v1:172.21.15.77:6811/2133267326] boot 2024-05-06T13:29:08.716 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:08 smithi077 ceph-mon[22428]: osdmap e13: 2 total, 2 up, 2 in 2024-05-06T13:29:08.716 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:08 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:29:08.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:08 smithi170 ceph-mon[23272]: osd.1 [v2:172.21.15.77:6810/2133267326,v1:172.21.15.77:6811/2133267326] boot 2024-05-06T13:29:08.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:08 smithi170 ceph-mon[23272]: osdmap e13: 2 total, 2 up, 2 in 2024-05-06T13:29:08.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:08 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:29:09.015 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi077:vg_nvme/lv_2 2024-05-06T13:29:09.757 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:09 smithi077 ceph-mon[22428]: osdmap e14: 2 total, 2 up, 2 in 2024-05-06T13:29:09.757 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:09 smithi077 ceph-mon[22428]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:09.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:09 smithi077 ceph-mon[27166]: osdmap e14: 2 total, 2 up, 2 in 2024-05-06T13:29:09.758 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:09 smithi077 ceph-mon[27166]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:09.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:09 smithi170 ceph-mon[23272]: osdmap e14: 2 total, 2 up, 2 in 2024-05-06T13:29:09.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:09 smithi170 ceph-mon[23272]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:10.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:10 smithi077 ceph-mon[22428]: Detected new or changed devices on smithi077 2024-05-06T13:29:10.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:10 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:10.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:10 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:29:10.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:10 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:29:10.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:10 smithi077 ceph-mon[22428]: Adjusting osd_memory_target on smithi077 to 8032M 2024-05-06T13:29:10.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:10 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:10.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:10 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:10.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:10 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:29:10.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:10 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:10.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:10 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:29:10.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:10 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:10.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:10 smithi077 ceph-mon[27166]: Detected new or changed devices on smithi077 2024-05-06T13:29:10.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:10 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:10.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:10 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:29:10.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:10 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:29:10.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:10 smithi077 ceph-mon[27166]: Adjusting osd_memory_target on smithi077 to 8032M 2024-05-06T13:29:10.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:10 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:10.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:10 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:10.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:10 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:29:10.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:10 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:10.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:10 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:29:10.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:10 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:11.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:10 smithi170 ceph-mon[23272]: Detected new or changed devices on smithi077 2024-05-06T13:29:11.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:10 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:11.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:10 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:29:11.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:10 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:29:11.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:10 smithi170 ceph-mon[23272]: Adjusting osd_memory_target on smithi077 to 8032M 2024-05-06T13:29:11.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:10 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:11.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:10 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:11.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:10 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:29:11.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:10 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:11.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:10 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:29:11.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:10 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:11.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:11 smithi077 ceph-mon[22428]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:11.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:11 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T13:29:11.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:11 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T13:29:11.745 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:11 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:11.745 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:11 smithi077 ceph-mon[27166]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:11.745 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:11 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T13:29:11.745 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:11 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T13:29:11.745 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:11 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:12.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:11 smithi170 ceph-mon[23272]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:12.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:11 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T13:29:12.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:11 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T13:29:12.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:11 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:12.904 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:12 smithi077 ceph-mon[22428]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:29:12.904 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:12 smithi077 ceph-mon[27166]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:29:13.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:12 smithi170 ceph-mon[23272]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:29:14.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:13 smithi077 ceph-mon[22428]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:14.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:13 smithi077 ceph-mon[27166]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:14.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:13 smithi170 ceph-mon[23272]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:15.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:14 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/174842408' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dcab9e69-bf02-4b9f-9ff1-5dbb5d71a557"}]: dispatch 2024-05-06T13:29:15.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:14 smithi077 ceph-mon[22428]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dcab9e69-bf02-4b9f-9ff1-5dbb5d71a557"}]: dispatch 2024-05-06T13:29:15.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:14 smithi077 ceph-mon[22428]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dcab9e69-bf02-4b9f-9ff1-5dbb5d71a557"}]': finished 2024-05-06T13:29:15.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:14 smithi077 ceph-mon[22428]: osdmap e15: 3 total, 2 up, 3 in 2024-05-06T13:29:15.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:14 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:29:15.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:14 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/4220435380' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T13:29:15.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:14 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/174842408' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dcab9e69-bf02-4b9f-9ff1-5dbb5d71a557"}]: dispatch 2024-05-06T13:29:15.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:14 smithi077 ceph-mon[27166]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dcab9e69-bf02-4b9f-9ff1-5dbb5d71a557"}]: dispatch 2024-05-06T13:29:15.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:14 smithi077 ceph-mon[27166]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dcab9e69-bf02-4b9f-9ff1-5dbb5d71a557"}]': finished 2024-05-06T13:29:15.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:14 smithi077 ceph-mon[27166]: osdmap e15: 3 total, 2 up, 3 in 2024-05-06T13:29:15.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:14 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:29:15.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:14 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/4220435380' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T13:29:15.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:14 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/174842408' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dcab9e69-bf02-4b9f-9ff1-5dbb5d71a557"}]: dispatch 2024-05-06T13:29:15.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:14 smithi170 ceph-mon[23272]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dcab9e69-bf02-4b9f-9ff1-5dbb5d71a557"}]: dispatch 2024-05-06T13:29:15.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:14 smithi170 ceph-mon[23272]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dcab9e69-bf02-4b9f-9ff1-5dbb5d71a557"}]': finished 2024-05-06T13:29:15.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:14 smithi170 ceph-mon[23272]: osdmap e15: 3 total, 2 up, 3 in 2024-05-06T13:29:15.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:14 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:29:15.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:14 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/4220435380' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T13:29:16.025 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:15 smithi170 ceph-mon[23272]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:16.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:15 smithi077 ceph-mon[22428]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:16.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:15 smithi077 ceph-mon[27166]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:17.798 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:17 smithi077 ceph-mon[22428]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:17.798 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:17 smithi077 ceph-mon[27166]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:18.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:17 smithi170 ceph-mon[23272]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:19.957 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:19 smithi077 ceph-mon[22428]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:19.958 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:19 smithi077 ceph-mon[27166]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:20.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:19 smithi170 ceph-mon[23272]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:21.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:20 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-06T13:29:21.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:20 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:21.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:20 smithi077 ceph-mon[22428]: Deploying daemon osd.2 on smithi077 2024-05-06T13:29:21.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:20 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-06T13:29:21.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:20 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:21.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:20 smithi077 ceph-mon[27166]: Deploying daemon osd.2 on smithi077 2024-05-06T13:29:21.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:20 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-06T13:29:21.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:20 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:21.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:20 smithi170 ceph-mon[23272]: Deploying daemon osd.2 on smithi077 2024-05-06T13:29:21.953 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:21 smithi077 ceph-mon[22428]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:21.953 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:21 smithi077 ceph-mon[27166]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:22.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:21 smithi170 ceph-mon[23272]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:24.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:23 smithi077 ceph-mon[27166]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:24.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:23 smithi077 ceph-mon[22428]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:24.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:23 smithi170 ceph-mon[23272]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:24.988 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:24 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:29:24.988 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:24 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:29:24.989 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:24 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:29:24.989 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:24 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:29:25.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:24 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:29:25.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:24 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:29:26.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:26 smithi077 ceph-mon[22428]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:26.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:26 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:26.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:26 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:29:26.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:26 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:26.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:26 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:29:26.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:26 smithi077 ceph-mon[27166]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:26.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:26 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:26.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:26 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:29:26.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:26 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:26.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:26 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:29:26.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:26 smithi170 ceph-mon[23272]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:26.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:26 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:26.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:26 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:29:26.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:26 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:26.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:26 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:29:28.141 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:28 smithi077 ceph-mon[22428]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:28.141 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:28 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:28.141 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:28 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:28.141 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:28 smithi077 ceph-mon[27166]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:28.141 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:28 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:28.141 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:28 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:28.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:28 smithi170 ceph-mon[23272]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:28.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:28 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:28.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:28 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:28.467 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 2 on host 'smithi077' 2024-05-06T13:29:29.081 DEBUG:teuthology.orchestra.run.smithi077:osd.2> sudo journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.2.service 2024-05-06T13:29:29.083 INFO:tasks.cephadm:Deploying osd.3 on smithi077 with /dev/vg_nvme/lv_1... 2024-05-06T13:29:29.084 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-06T13:29:29.293 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:29 smithi077 ceph-mon[22428]: from='osd.2 [v2:172.21.15.77:6818/2522642178,v1:172.21.15.77:6819/2522642178]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-06T13:29:29.293 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:29 smithi077 ceph-mon[22428]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-06T13:29:29.294 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:29 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:29.294 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:29 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:29:29.294 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:29 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:29.294 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:29 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:29:29.294 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:29 smithi077 ceph-mon[27166]: from='osd.2 [v2:172.21.15.77:6818/2522642178,v1:172.21.15.77:6819/2522642178]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-06T13:29:29.294 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:29 smithi077 ceph-mon[27166]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-06T13:29:29.294 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:29 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:29.294 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:29 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:29:29.294 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:29 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:29.294 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:29 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:29:29.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:29 smithi170 ceph-mon[23272]: from='osd.2 [v2:172.21.15.77:6818/2522642178,v1:172.21.15.77:6819/2522642178]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-06T13:29:29.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:29 smithi170 ceph-mon[23272]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-06T13:29:29.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:29 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:29.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:29 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:29:29.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:29 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:29.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:29 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:29:30.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:30 smithi077 ceph-mon[22428]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:30.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:30 smithi077 ceph-mon[22428]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-06T13:29:30.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:30 smithi077 ceph-mon[22428]: osdmap e16: 3 total, 2 up, 3 in 2024-05-06T13:29:30.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:30 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:29:30.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:30 smithi077 ceph-mon[22428]: from='osd.2 [v2:172.21.15.77:6818/2522642178,v1:172.21.15.77:6819/2522642178]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T13:29:30.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:30 smithi077 ceph-mon[22428]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T13:29:30.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:30 smithi077 ceph-mon[27166]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:30.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:30 smithi077 ceph-mon[27166]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-06T13:29:30.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:30 smithi077 ceph-mon[27166]: osdmap e16: 3 total, 2 up, 3 in 2024-05-06T13:29:30.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:30 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:29:30.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:30 smithi077 ceph-mon[27166]: from='osd.2 [v2:172.21.15.77:6818/2522642178,v1:172.21.15.77:6819/2522642178]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T13:29:30.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:30 smithi077 ceph-mon[27166]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T13:29:30.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 13:29:30 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2[35912]: 2024-05-06T13:29:30.050+0000 7f0162648700 -1 osd.2 0 waiting for initial osdmap 2024-05-06T13:29:30.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 13:29:30 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2[35912]: 2024-05-06T13:29:30.053+0000 7f015c7de700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T13:29:30.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:30 smithi170 ceph-mon[23272]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:30.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:30 smithi170 ceph-mon[23272]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-06T13:29:30.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:30 smithi170 ceph-mon[23272]: osdmap e16: 3 total, 2 up, 3 in 2024-05-06T13:29:30.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:30 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:29:30.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:30 smithi170 ceph-mon[23272]: from='osd.2 [v2:172.21.15.77:6818/2522642178,v1:172.21.15.77:6819/2522642178]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T13:29:30.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:30 smithi170 ceph-mon[23272]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T13:29:31.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:31 smithi077 ceph-mon[22428]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-05-06T13:29:31.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:31 smithi077 ceph-mon[22428]: osdmap e17: 3 total, 2 up, 3 in 2024-05-06T13:29:31.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:31 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:29:31.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:31 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:29:31.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:31 smithi077 ceph-mon[27166]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-05-06T13:29:31.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:31 smithi077 ceph-mon[27166]: osdmap e17: 3 total, 2 up, 3 in 2024-05-06T13:29:31.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:31 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:29:31.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:31 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:29:31.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:31 smithi170 ceph-mon[23272]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-05-06T13:29:31.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:31 smithi170 ceph-mon[23272]: osdmap e17: 3 total, 2 up, 3 in 2024-05-06T13:29:31.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:31 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:29:31.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:31 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:29:32.196 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:32 smithi077 ceph-mon[22428]: purged_snaps scrub starts 2024-05-06T13:29:32.196 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:32 smithi077 ceph-mon[22428]: purged_snaps scrub ok 2024-05-06T13:29:32.197 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:32 smithi077 ceph-mon[22428]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:32.197 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:32 smithi077 ceph-mon[22428]: osd.2 [v2:172.21.15.77:6818/2522642178,v1:172.21.15.77:6819/2522642178] boot 2024-05-06T13:29:32.197 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:32 smithi077 ceph-mon[22428]: osdmap e18: 3 total, 3 up, 3 in 2024-05-06T13:29:32.197 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:32 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:29:32.197 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:32 smithi077 ceph-mon[27166]: purged_snaps scrub starts 2024-05-06T13:29:32.197 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:32 smithi077 ceph-mon[27166]: purged_snaps scrub ok 2024-05-06T13:29:32.197 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:32 smithi077 ceph-mon[27166]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:32.197 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:32 smithi077 ceph-mon[27166]: osd.2 [v2:172.21.15.77:6818/2522642178,v1:172.21.15.77:6819/2522642178] boot 2024-05-06T13:29:32.197 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:32 smithi077 ceph-mon[27166]: osdmap e18: 3 total, 3 up, 3 in 2024-05-06T13:29:32.197 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:32 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:29:32.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:32 smithi170 ceph-mon[23272]: purged_snaps scrub starts 2024-05-06T13:29:32.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:32 smithi170 ceph-mon[23272]: purged_snaps scrub ok 2024-05-06T13:29:32.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:32 smithi170 ceph-mon[23272]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-05-06T13:29:32.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:32 smithi170 ceph-mon[23272]: osd.2 [v2:172.21.15.77:6818/2522642178,v1:172.21.15.77:6819/2522642178] boot 2024-05-06T13:29:32.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:32 smithi170 ceph-mon[23272]: osdmap e18: 3 total, 3 up, 3 in 2024-05-06T13:29:32.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:32 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:29:32.572 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-06T13:29:32.572 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-06T13:29:32.572 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-05-06T13:29:32.572 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-05-06T13:29:32.572 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0163811 s, 640 MB/s 2024-05-06T13:29:32.572 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-05-06T13:29:33.321 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:33 smithi077 ceph-mon[22428]: osdmap e19: 3 total, 3 up, 3 in 2024-05-06T13:29:33.321 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:33 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-05-06T13:29:33.321 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:33 smithi077 ceph-mon[27166]: osdmap e19: 3 total, 3 up, 3 in 2024-05-06T13:29:33.321 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:33 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-05-06T13:29:33.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:33 smithi170 ceph-mon[23272]: osdmap e19: 3 total, 3 up, 3 in 2024-05-06T13:29:33.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:33 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-05-06T13:29:33.887 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi077:vg_nvme/lv_1 2024-05-06T13:29:34.189 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:34 smithi077 ceph-mon[22428]: pgmap v73: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:34.190 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:34 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-05-06T13:29:34.190 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:34 smithi077 ceph-mon[22428]: osdmap e20: 3 total, 3 up, 3 in 2024-05-06T13:29:34.190 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:34 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-06T13:29:34.190 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:34 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:34.190 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:34 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:29:34.190 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:34 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:29:34.190 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:34 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:29:34.190 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:34 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:34.190 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:34 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:34.190 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:34 smithi077 ceph-mon[27166]: pgmap v73: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:34.191 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:34 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-05-06T13:29:34.191 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:34 smithi077 ceph-mon[27166]: osdmap e20: 3 total, 3 up, 3 in 2024-05-06T13:29:34.191 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:34 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-06T13:29:34.191 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:34 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:34.191 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:34 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:29:34.191 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:34 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:29:34.191 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:34 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:29:34.191 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:34 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:34.191 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:34 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:34.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:34 smithi170 ceph-mon[23272]: pgmap v73: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:34.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:34 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-05-06T13:29:34.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:34 smithi170 ceph-mon[23272]: osdmap e20: 3 total, 3 up, 3 in 2024-05-06T13:29:34.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:34 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-06T13:29:34.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:34 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:34.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:34 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:29:34.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:34 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:29:34.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:34 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:29:34.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:34 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:34.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:34 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:34.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:29:34 smithi077 sudo[37203]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-06T13:29:34.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:29:34 smithi077 sudo[37203]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T13:29:34.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:29:34 smithi077 sudo[37203]: pam_unix(sudo:session): session closed for user root 2024-05-06T13:29:35.220 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:29:34 smithi077 sudo[37209]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-06T13:29:35.221 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:29:34 smithi077 sudo[37209]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T13:29:35.221 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:29:34 smithi077 sudo[37209]: pam_unix(sudo:session): session closed for user root 2024-05-06T13:29:35.221 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:35 smithi077 ceph-mon[22428]: Detected new or changed devices on smithi077 2024-05-06T13:29:35.221 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:35 smithi077 ceph-mon[22428]: Adjusting osd_memory_target on smithi077 to 5354M 2024-05-06T13:29:35.221 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:35 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-06T13:29:35.221 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:35 smithi077 ceph-mon[22428]: osdmap e21: 3 total, 3 up, 3 in 2024-05-06T13:29:35.221 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:35 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:35.221 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:35 smithi077 ceph-mon[27166]: Detected new or changed devices on smithi077 2024-05-06T13:29:35.222 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:35 smithi077 ceph-mon[27166]: Adjusting osd_memory_target on smithi077 to 5354M 2024-05-06T13:29:35.222 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:35 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-06T13:29:35.222 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:35 smithi077 ceph-mon[27166]: osdmap e21: 3 total, 3 up, 3 in 2024-05-06T13:29:35.222 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:35 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:35.477 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 13:29:35 smithi077 sudo[37212]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-06T13:29:35.477 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 13:29:35 smithi077 sudo[37212]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T13:29:35.477 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 13:29:35 smithi077 sudo[37212]: pam_unix(sudo:session): session closed for user root 2024-05-06T13:29:35.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:35 smithi170 ceph-mon[23272]: Detected new or changed devices on smithi077 2024-05-06T13:29:35.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:35 smithi170 ceph-mon[23272]: Adjusting osd_memory_target on smithi077 to 5354M 2024-05-06T13:29:35.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:35 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-06T13:29:35.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:35 smithi170 ceph-mon[23272]: osdmap e21: 3 total, 3 up, 3 in 2024-05-06T13:29:35.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:35 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:35.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 13:29:35 smithi077 sudo[37235]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-06T13:29:35.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 13:29:35 smithi077 sudo[37235]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T13:29:35.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 13:29:35 smithi077 sudo[37235]: pam_unix(sudo:session): session closed for user root 2024-05-06T13:29:35.997 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 13:29:35 smithi077 sudo[37263]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-06T13:29:35.997 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 13:29:35 smithi077 sudo[37263]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T13:29:35.998 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 13:29:35 smithi077 sudo[37263]: pam_unix(sudo:session): session closed for user root 2024-05-06T13:29:36.254 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 13:29:35 smithi077 sudo[37303]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-06T13:29:36.254 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 13:29:35 smithi077 sudo[37303]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T13:29:36.255 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 13:29:36 smithi077 sudo[37303]: pam_unix(sudo:session): session closed for user root 2024-05-06T13:29:36.506 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:36 smithi077 sudo[37346]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-06T13:29:36.506 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:36 smithi077 sudo[37346]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T13:29:36.506 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:36 smithi077 sudo[37346]: pam_unix(sudo:session): session closed for user root 2024-05-06T13:29:36.765 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:36 smithi077 ceph-mon[27166]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:36.765 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:36 smithi077 ceph-mon[27166]: osdmap e22: 3 total, 3 up, 3 in 2024-05-06T13:29:36.765 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:36 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T13:29:36.765 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:36 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T13:29:36.765 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:36 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:36.765 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:36 smithi077 ceph-mon[27166]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-06T13:29:36.766 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:36 smithi077 sudo[37355]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-06T13:29:36.766 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:36 smithi077 sudo[37355]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T13:29:36.766 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:36 smithi077 sudo[37355]: pam_unix(sudo:session): session closed for user root 2024-05-06T13:29:36.766 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:36 smithi077 ceph-mon[22428]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:36.766 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:36 smithi077 ceph-mon[22428]: osdmap e22: 3 total, 3 up, 3 in 2024-05-06T13:29:36.766 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:36 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T13:29:36.766 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:36 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T13:29:36.766 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:36 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:36.766 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:36 smithi077 ceph-mon[22428]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-06T13:29:36.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:36 smithi170 ceph-mon[23272]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:36.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:36 smithi170 ceph-mon[23272]: osdmap e22: 3 total, 3 up, 3 in 2024-05-06T13:29:36.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:36 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T13:29:36.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:36 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T13:29:36.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:36 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:36.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:36 smithi170 ceph-mon[23272]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-06T13:29:37.019 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:36 smithi077 sudo[37369]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-06T13:29:37.019 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:36 smithi077 sudo[37369]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T13:29:37.019 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:36 smithi077 sudo[37369]: pam_unix(sudo:session): session closed for user root 2024-05-06T13:29:37.295 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:37 smithi077 sudo[37375]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-06T13:29:37.295 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:37 smithi077 sudo[37375]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T13:29:37.295 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:37 smithi077 sudo[37375]: pam_unix(sudo:session): session closed for user root 2024-05-06T13:29:37.516 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:37 smithi170 sudo[24787]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-06T13:29:37.516 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:37 smithi170 sudo[24787]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T13:29:37.516 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:37 smithi170 sudo[24787]: pam_unix(sudo:session): session closed for user root 2024-05-06T13:29:37.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[22428]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:29:37.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[22428]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:37.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[22428]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-06T13:29:37.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:29:37.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:29:37.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:29:37.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[22428]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-06T13:29:37.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:29:37.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:29:37.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:29:37.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[22428]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-06T13:29:37.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[22428]: mgrmap e15: y(active, since 2m), standbys: x 2024-05-06T13:29:37.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:29:37.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:29:37.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:29:37.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[27166]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:29:37.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[27166]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:37.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[27166]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-06T13:29:37.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:29:37.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:29:37.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:29:37.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[27166]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-06T13:29:37.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:29:37.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:29:37.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:29:37.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[27166]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-06T13:29:37.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[27166]: mgrmap e15: y(active, since 2m), standbys: x 2024-05-06T13:29:37.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:29:37.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:29:37.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:37 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:29:37.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:37 smithi170 sudo[24790]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-06T13:29:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:37 smithi170 sudo[24790]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-05-06T13:29:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:37 smithi170 sudo[24790]: pam_unix(sudo:session): session closed for user root 2024-05-06T13:29:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:37 smithi170 ceph-mon[23272]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:29:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:37 smithi170 ceph-mon[23272]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:37 smithi170 ceph-mon[23272]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-06T13:29:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:37 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:29:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:37 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:29:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:37 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:29:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:37 smithi170 ceph-mon[23272]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-06T13:29:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:37 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:29:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:37 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:29:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:37 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:29:37.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:37 smithi170 ceph-mon[23272]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-06T13:29:37.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:37 smithi170 ceph-mon[23272]: mgrmap e15: y(active, since 2m), standbys: x 2024-05-06T13:29:37.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:37 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:29:37.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:37 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:29:37.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:37 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:29:38.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:38 smithi077 ceph-mon[22428]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-06T13:29:38.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:38 smithi077 ceph-mon[22428]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-06T13:29:38.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:38 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:38.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:38 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3822369766' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b177913-e714-4f93-99b4-e2da20777d59"}]: dispatch 2024-05-06T13:29:38.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:38 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3822369766' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b177913-e714-4f93-99b4-e2da20777d59"}]': finished 2024-05-06T13:29:38.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:38 smithi077 ceph-mon[22428]: osdmap e23: 4 total, 3 up, 4 in 2024-05-06T13:29:38.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:38 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:29:38.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:38 smithi077 ceph-mon[27166]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-06T13:29:38.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:38 smithi077 ceph-mon[27166]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-06T13:29:38.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:38 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:38.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:38 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3822369766' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b177913-e714-4f93-99b4-e2da20777d59"}]: dispatch 2024-05-06T13:29:38.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:38 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3822369766' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b177913-e714-4f93-99b4-e2da20777d59"}]': finished 2024-05-06T13:29:38.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:38 smithi077 ceph-mon[27166]: osdmap e23: 4 total, 3 up, 4 in 2024-05-06T13:29:38.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:38 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:29:38.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:38 smithi170 ceph-mon[23272]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-06T13:29:38.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:38 smithi170 ceph-mon[23272]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-06T13:29:38.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:38 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:38.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:38 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/3822369766' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b177913-e714-4f93-99b4-e2da20777d59"}]: dispatch 2024-05-06T13:29:38.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:38 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/3822369766' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b177913-e714-4f93-99b4-e2da20777d59"}]': finished 2024-05-06T13:29:38.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:38 smithi170 ceph-mon[23272]: osdmap e23: 4 total, 3 up, 4 in 2024-05-06T13:29:38.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:38 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:29:39.801 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:39 smithi077 ceph-mon[27166]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:39.801 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:39 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3519913210' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T13:29:39.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:39 smithi077 ceph-mon[22428]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:39.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:39 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3519913210' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T13:29:39.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:39 smithi170 ceph-mon[23272]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:39.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:39 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/3519913210' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T13:29:41.601 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:41 smithi077 ceph-mon[22428]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:41.601 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:41 smithi077 ceph-mon[27166]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:41.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:41 smithi170 ceph-mon[23272]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:43.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:43 smithi077 ceph-mon[22428]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:43.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:43 smithi077 ceph-mon[27166]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:43.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:43 smithi170 ceph-mon[23272]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:45.790 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:45 smithi077 ceph-mon[27166]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:45.790 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:45 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-06T13:29:45.790 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:45 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:45.790 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:45 smithi077 ceph-mon[27166]: Deploying daemon osd.3 on smithi077 2024-05-06T13:29:45.790 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:45 smithi077 ceph-mon[22428]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:45.790 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:45 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-06T13:29:45.790 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:45 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:45.790 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:45 smithi077 ceph-mon[22428]: Deploying daemon osd.3 on smithi077 2024-05-06T13:29:45.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:45 smithi170 ceph-mon[23272]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:45.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:45 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-06T13:29:45.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:45 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:45.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:45 smithi170 ceph-mon[23272]: Deploying daemon osd.3 on smithi077 2024-05-06T13:29:47.776 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:47 smithi077 ceph-mon[22428]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:47.777 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:47 smithi077 ceph-mon[27166]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:47.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:47 smithi170 ceph-mon[23272]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:49.761 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:49 smithi077 ceph-mon[27166]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:49.761 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:49 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:49.761 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:49 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:29:49.761 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:49 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:49.761 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:49 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:29:49.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:49 smithi077 ceph-mon[22428]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:49.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:49 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:49.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:49 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:29:49.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:49 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:49.762 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:49 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:29:49.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:49 smithi170 ceph-mon[23272]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:49.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:49 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:49.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:49 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:29:49.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:49 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:49.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:49 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:29:51.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:51 smithi077 ceph-mon[22428]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:51.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:51 smithi077 ceph-mon[27166]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:51.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:51 smithi170 ceph-mon[23272]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:52.933 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 3 on host 'smithi077' 2024-05-06T13:29:53.232 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:53 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:53.232 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:53 smithi077 ceph-mon[22428]: from='osd.3 [v2:172.21.15.77:6826/3252753833,v1:172.21.15.77:6827/3252753833]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-06T13:29:53.232 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:53 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:53.232 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:53 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:53.232 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:53 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:29:53.232 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:53 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:53.232 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:53 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:29:53.233 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:53 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:53.233 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:53 smithi077 ceph-mon[27166]: from='osd.3 [v2:172.21.15.77:6826/3252753833,v1:172.21.15.77:6827/3252753833]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-06T13:29:53.233 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:53 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:53.233 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:53 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:53.233 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:53 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:29:53.233 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:53 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:53.233 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:53 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:29:53.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:53 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:53.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:53 smithi170 ceph-mon[23272]: from='osd.3 [v2:172.21.15.77:6826/3252753833,v1:172.21.15.77:6827/3252753833]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-06T13:29:53.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:53 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:53.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:53 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:53.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:53 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:29:53.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:53 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:53.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:53 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:29:53.555 DEBUG:teuthology.orchestra.run.smithi077:osd.3> sudo journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.3.service 2024-05-06T13:29:53.558 INFO:tasks.cephadm:Deploying osd.4 on smithi170 with /dev/vg_nvme/lv_4... 2024-05-06T13:29:53.558 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-06T13:29:54.122 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:54 smithi170 ceph-mon[23272]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:54.123 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:54 smithi170 ceph-mon[23272]: from='osd.3 [v2:172.21.15.77:6826/3252753833,v1:172.21.15.77:6827/3252753833]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-06T13:29:54.123 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:54 smithi170 ceph-mon[23272]: osdmap e24: 4 total, 3 up, 4 in 2024-05-06T13:29:54.123 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:54 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:29:54.123 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:54 smithi170 ceph-mon[23272]: from='osd.3 [v2:172.21.15.77:6826/3252753833,v1:172.21.15.77:6827/3252753833]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T13:29:54.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:54 smithi077 ceph-mon[22428]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:54.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:54 smithi077 ceph-mon[22428]: from='osd.3 [v2:172.21.15.77:6826/3252753833,v1:172.21.15.77:6827/3252753833]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-06T13:29:54.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:54 smithi077 ceph-mon[22428]: osdmap e24: 4 total, 3 up, 4 in 2024-05-06T13:29:54.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:54 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:29:54.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:54 smithi077 ceph-mon[22428]: from='osd.3 [v2:172.21.15.77:6826/3252753833,v1:172.21.15.77:6827/3252753833]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T13:29:54.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:54 smithi077 ceph-mon[27166]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:54.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:54 smithi077 ceph-mon[27166]: from='osd.3 [v2:172.21.15.77:6826/3252753833,v1:172.21.15.77:6827/3252753833]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-06T13:29:54.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:54 smithi077 ceph-mon[27166]: osdmap e24: 4 total, 3 up, 4 in 2024-05-06T13:29:54.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:54 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:29:54.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:54 smithi077 ceph-mon[27166]: from='osd.3 [v2:172.21.15.77:6826/3252753833,v1:172.21.15.77:6827/3252753833]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T13:29:54.303 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 13:29:54 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3[38889]: 2024-05-06T13:29:54.054+0000 7f260b059700 -1 osd.3 0 waiting for initial osdmap 2024-05-06T13:29:54.303 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 13:29:54 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3[38889]: 2024-05-06T13:29:54.063+0000 7f26061f1700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T13:29:55.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:55 smithi170 ceph-mon[23272]: from='osd.3 [v2:172.21.15.77:6826/3252753833,v1:172.21.15.77:6827/3252753833]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-05-06T13:29:55.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:55 smithi170 ceph-mon[23272]: osdmap e25: 4 total, 3 up, 4 in 2024-05-06T13:29:55.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:55 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:29:55.132 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:55 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:29:55.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:55 smithi077 ceph-mon[22428]: from='osd.3 [v2:172.21.15.77:6826/3252753833,v1:172.21.15.77:6827/3252753833]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-05-06T13:29:55.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:55 smithi077 ceph-mon[22428]: osdmap e25: 4 total, 3 up, 4 in 2024-05-06T13:29:55.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:55 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:29:55.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:55 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:29:55.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:55 smithi077 ceph-mon[27166]: from='osd.3 [v2:172.21.15.77:6826/3252753833,v1:172.21.15.77:6827/3252753833]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-05-06T13:29:55.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:55 smithi077 ceph-mon[27166]: osdmap e25: 4 total, 3 up, 4 in 2024-05-06T13:29:55.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:55 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:29:55.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:55 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:29:55.608 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-06T13:29:55.609 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-06T13:29:55.609 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-05-06T13:29:55.609 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-05-06T13:29:55.609 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0087728 s, 1.2 GB/s 2024-05-06T13:29:55.609 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-05-06T13:29:56.218 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:56 smithi170 ceph-mon[23272]: purged_snaps scrub starts 2024-05-06T13:29:56.218 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:56 smithi170 ceph-mon[23272]: purged_snaps scrub ok 2024-05-06T13:29:56.218 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:56 smithi170 ceph-mon[23272]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:56.219 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:56 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:29:56.219 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:56 smithi170 ceph-mon[23272]: osd.3 [v2:172.21.15.77:6826/3252753833,v1:172.21.15.77:6827/3252753833] boot 2024-05-06T13:29:56.219 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:56 smithi170 ceph-mon[23272]: osdmap e26: 4 total, 4 up, 4 in 2024-05-06T13:29:56.219 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:56 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:29:56.220 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi170:vg_nvme/lv_4 2024-05-06T13:29:56.415 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:56 smithi077 ceph-mon[22428]: purged_snaps scrub starts 2024-05-06T13:29:56.415 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:56 smithi077 ceph-mon[22428]: purged_snaps scrub ok 2024-05-06T13:29:56.415 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:56 smithi077 ceph-mon[22428]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:56.415 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:56 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:29:56.415 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:56 smithi077 ceph-mon[22428]: osd.3 [v2:172.21.15.77:6826/3252753833,v1:172.21.15.77:6827/3252753833] boot 2024-05-06T13:29:56.415 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:56 smithi077 ceph-mon[22428]: osdmap e26: 4 total, 4 up, 4 in 2024-05-06T13:29:56.415 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:56 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:29:56.415 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:56 smithi077 ceph-mon[27166]: purged_snaps scrub starts 2024-05-06T13:29:56.415 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:56 smithi077 ceph-mon[27166]: purged_snaps scrub ok 2024-05-06T13:29:56.415 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:56 smithi077 ceph-mon[27166]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-05-06T13:29:56.415 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:56 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:29:56.416 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:56 smithi077 ceph-mon[27166]: osd.3 [v2:172.21.15.77:6826/3252753833,v1:172.21.15.77:6827/3252753833] boot 2024-05-06T13:29:56.416 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:56 smithi077 ceph-mon[27166]: osdmap e26: 4 total, 4 up, 4 in 2024-05-06T13:29:56.416 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:56 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:29:57.168 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:57 smithi170 ceph-mon[23272]: osdmap e27: 4 total, 4 up, 4 in 2024-05-06T13:29:57.168 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:57 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:57.168 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:57 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:29:57.168 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:57 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:29:57.168 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:57 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:29:57.168 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:57 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:29:57.168 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:57 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:57.168 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:57 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:57.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:57 smithi077 ceph-mon[22428]: osdmap e27: 4 total, 4 up, 4 in 2024-05-06T13:29:57.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:57 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:57.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:57 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:29:57.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:57 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:29:57.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:57 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:29:57.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:57 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:29:57.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:57 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:57.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:57 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:57.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:57 smithi077 ceph-mon[27166]: osdmap e27: 4 total, 4 up, 4 in 2024-05-06T13:29:57.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:57 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:57.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:57 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:29:57.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:57 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:29:57.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:57 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:29:57.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:57 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:29:57.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:57 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:57.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:57 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:29:58.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:58 smithi170 ceph-mon[23272]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:29:58.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:58 smithi170 ceph-mon[23272]: Detected new or changed devices on smithi077 2024-05-06T13:29:58.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:58 smithi170 ceph-mon[23272]: Adjusting osd_memory_target on smithi077 to 4016M 2024-05-06T13:29:58.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:58 smithi077 ceph-mon[22428]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:29:58.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:58 smithi077 ceph-mon[22428]: Detected new or changed devices on smithi077 2024-05-06T13:29:58.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:58 smithi077 ceph-mon[22428]: Adjusting osd_memory_target on smithi077 to 4016M 2024-05-06T13:29:58.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:58 smithi077 ceph-mon[27166]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:29:58.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:58 smithi077 ceph-mon[27166]: Detected new or changed devices on smithi077 2024-05-06T13:29:58.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:58 smithi077 ceph-mon[27166]: Adjusting osd_memory_target on smithi077 to 4016M 2024-05-06T13:29:59.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:59 smithi170 ceph-mon[23272]: from='client.34193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:29:59.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:59 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T13:29:59.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:59 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T13:29:59.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:29:59 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:59.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:59 smithi077 ceph-mon[22428]: from='client.34193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:29:59.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:59 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T13:29:59.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:59 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T13:29:59.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:29:59 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:29:59.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:59 smithi077 ceph-mon[27166]: from='client.34193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:29:59.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:59 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T13:29:59.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:59 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T13:29:59.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:29:59 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:00.528 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:00 smithi170 ceph-mon[23272]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:00.528 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:00 smithi170 ceph-mon[23272]: overall HEALTH_OK 2024-05-06T13:30:00.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:00 smithi077 ceph-mon[22428]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:00.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:00 smithi077 ceph-mon[22428]: overall HEALTH_OK 2024-05-06T13:30:00.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:00 smithi077 ceph-mon[27166]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:00.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:00 smithi077 ceph-mon[27166]: overall HEALTH_OK 2024-05-06T13:30:01.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:01 smithi077 ceph-mon[22428]: from='client.? 172.21.15.170:0/2582685959' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce6d16fa-5c8c-4a94-852b-f8193d47d25f"}]: dispatch 2024-05-06T13:30:01.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:01 smithi077 ceph-mon[22428]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce6d16fa-5c8c-4a94-852b-f8193d47d25f"}]: dispatch 2024-05-06T13:30:01.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:01 smithi077 ceph-mon[22428]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce6d16fa-5c8c-4a94-852b-f8193d47d25f"}]': finished 2024-05-06T13:30:01.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:01 smithi077 ceph-mon[22428]: osdmap e28: 5 total, 4 up, 5 in 2024-05-06T13:30:01.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:01 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:30:01.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:01 smithi077 ceph-mon[22428]: from='client.? 172.21.15.170:0/1880513613' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T13:30:01.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:01 smithi077 ceph-mon[27166]: from='client.? 172.21.15.170:0/2582685959' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce6d16fa-5c8c-4a94-852b-f8193d47d25f"}]: dispatch 2024-05-06T13:30:01.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:01 smithi077 ceph-mon[27166]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce6d16fa-5c8c-4a94-852b-f8193d47d25f"}]: dispatch 2024-05-06T13:30:01.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:01 smithi077 ceph-mon[27166]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce6d16fa-5c8c-4a94-852b-f8193d47d25f"}]': finished 2024-05-06T13:30:01.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:01 smithi077 ceph-mon[27166]: osdmap e28: 5 total, 4 up, 5 in 2024-05-06T13:30:01.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:01 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:30:01.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:01 smithi077 ceph-mon[27166]: from='client.? 172.21.15.170:0/1880513613' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T13:30:01.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:01 smithi170 ceph-mon[23272]: from='client.? 172.21.15.170:0/2582685959' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce6d16fa-5c8c-4a94-852b-f8193d47d25f"}]: dispatch 2024-05-06T13:30:01.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:01 smithi170 ceph-mon[23272]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce6d16fa-5c8c-4a94-852b-f8193d47d25f"}]: dispatch 2024-05-06T13:30:01.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:01 smithi170 ceph-mon[23272]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce6d16fa-5c8c-4a94-852b-f8193d47d25f"}]': finished 2024-05-06T13:30:01.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:01 smithi170 ceph-mon[23272]: osdmap e28: 5 total, 4 up, 5 in 2024-05-06T13:30:01.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:01 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:30:01.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:01 smithi170 ceph-mon[23272]: from='client.? 172.21.15.170:0/1880513613' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T13:30:02.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:02 smithi077 ceph-mon[27166]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:02.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:02 smithi077 ceph-mon[22428]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:02.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:02 smithi170 ceph-mon[23272]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:04.323 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:04 smithi170 ceph-mon[23272]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:04.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:04 smithi077 ceph-mon[22428]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:04.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:04 smithi077 ceph-mon[27166]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:06.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:06 smithi170 ceph-mon[23272]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:06.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:06 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-06T13:30:06.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:06 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:06.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:06 smithi077 ceph-mon[22428]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:06.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:06 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-06T13:30:06.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:06 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:06.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:06 smithi077 ceph-mon[27166]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:06.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:06 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-06T13:30:06.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:06 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:07.347 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:07 smithi170 ceph-mon[23272]: Deploying daemon osd.4 on smithi170 2024-05-06T13:30:07.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:07 smithi077 ceph-mon[27166]: Deploying daemon osd.4 on smithi170 2024-05-06T13:30:07.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:07 smithi077 ceph-mon[22428]: Deploying daemon osd.4 on smithi170 2024-05-06T13:30:08.513 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:08 smithi170 ceph-mon[23272]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:08.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:08 smithi077 ceph-mon[27166]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:08.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:08 smithi077 ceph-mon[22428]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:10.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:10 smithi170 ceph-mon[23272]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:10.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:10 smithi077 ceph-mon[22428]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:10.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:10 smithi077 ceph-mon[27166]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:12.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:11 smithi077 ceph-mon[27166]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:12.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:11 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:12.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:11 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:30:12.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:11 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:12.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:11 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:30:12.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:11 smithi077 ceph-mon[22428]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:12.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:11 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:12.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:11 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:30:12.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:11 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:12.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:11 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:30:12.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:11 smithi170 ceph-mon[23272]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:12.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:11 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:12.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:11 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:30:12.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:11 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:12.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:11 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:30:13.856 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 4 on host 'smithi170' 2024-05-06T13:30:14.172 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:13 smithi170 ceph-mon[23272]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:14.172 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:13 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:14.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:13 smithi077 ceph-mon[22428]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:14.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:13 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:14.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:13 smithi077 ceph-mon[27166]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:14.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:13 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:14.478 DEBUG:teuthology.orchestra.run.smithi170:osd.4> sudo journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.4.service 2024-05-06T13:30:14.481 INFO:tasks.cephadm:Deploying osd.5 on smithi170 with /dev/vg_nvme/lv_3... 2024-05-06T13:30:14.481 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-06T13:30:15.066 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:14 smithi170 ceph-mon[23272]: from='osd.4 [v2:172.21.15.170:6800/4006535878,v1:172.21.15.170:6801/4006535878]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T13:30:15.066 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:14 smithi170 ceph-mon[23272]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T13:30:15.066 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:14 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:15.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:14 smithi077 ceph-mon[22428]: from='osd.4 [v2:172.21.15.170:6800/4006535878,v1:172.21.15.170:6801/4006535878]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T13:30:15.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:14 smithi077 ceph-mon[22428]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T13:30:15.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:14 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:15.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:14 smithi077 ceph-mon[27166]: from='osd.4 [v2:172.21.15.170:6800/4006535878,v1:172.21.15.170:6801/4006535878]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T13:30:15.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:14 smithi077 ceph-mon[27166]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T13:30:15.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:14 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:16.059 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:15 smithi170 ceph-mon[23272]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:16.059 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:15 smithi170 ceph-mon[23272]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-06T13:30:16.059 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:15 smithi170 ceph-mon[23272]: osdmap e29: 5 total, 4 up, 5 in 2024-05-06T13:30:16.059 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:15 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:30:16.059 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:15 smithi170 ceph-mon[23272]: from='osd.4 [v2:172.21.15.170:6800/4006535878,v1:172.21.15.170:6801/4006535878]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T13:30:16.059 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:15 smithi170 ceph-mon[23272]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T13:30:16.060 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 13:30:15 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4[26640]: 2024-05-06T13:30:15.966+0000 7f767e95a700 -1 osd.4 0 waiting for initial osdmap 2024-05-06T13:30:16.060 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 13:30:15 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4[26640]: 2024-05-06T13:30:15.970+0000 7f767aaf4700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T13:30:16.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:15 smithi077 ceph-mon[22428]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:16.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:15 smithi077 ceph-mon[22428]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-06T13:30:16.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:15 smithi077 ceph-mon[22428]: osdmap e29: 5 total, 4 up, 5 in 2024-05-06T13:30:16.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:15 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:30:16.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:15 smithi077 ceph-mon[22428]: from='osd.4 [v2:172.21.15.170:6800/4006535878,v1:172.21.15.170:6801/4006535878]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T13:30:16.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:15 smithi077 ceph-mon[22428]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T13:30:16.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:15 smithi077 ceph-mon[27166]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:16.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:15 smithi077 ceph-mon[27166]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-06T13:30:16.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:15 smithi077 ceph-mon[27166]: osdmap e29: 5 total, 4 up, 5 in 2024-05-06T13:30:16.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:15 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:30:16.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:15 smithi077 ceph-mon[27166]: from='osd.4 [v2:172.21.15.170:6800/4006535878,v1:172.21.15.170:6801/4006535878]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T13:30:16.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:15 smithi077 ceph-mon[27166]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T13:30:16.560 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-06T13:30:16.561 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-06T13:30:16.561 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-05-06T13:30:16.561 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-05-06T13:30:16.561 INFO:teuthology.orchestra.run.smithi170.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0100748 s, 1.0 GB/s 2024-05-06T13:30:16.561 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 2024-05-06T13:30:16.561 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-05-06T13:30:17.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:16 smithi170 ceph-mon[23272]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-05-06T13:30:17.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:16 smithi170 ceph-mon[23272]: osdmap e30: 5 total, 4 up, 5 in 2024-05-06T13:30:17.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:16 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:30:17.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:16 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:30:17.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:16 smithi077 ceph-mon[22428]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-05-06T13:30:17.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:16 smithi077 ceph-mon[22428]: osdmap e30: 5 total, 4 up, 5 in 2024-05-06T13:30:17.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:16 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:30:17.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:16 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:30:17.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:16 smithi077 ceph-mon[27166]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-05-06T13:30:17.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:16 smithi077 ceph-mon[27166]: osdmap e30: 5 total, 4 up, 5 in 2024-05-06T13:30:17.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:16 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:30:17.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:16 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:30:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:17 smithi077 ceph-mon[22428]: purged_snaps scrub starts 2024-05-06T13:30:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:17 smithi077 ceph-mon[22428]: purged_snaps scrub ok 2024-05-06T13:30:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:17 smithi077 ceph-mon[22428]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:17 smithi077 ceph-mon[22428]: osd.4 [v2:172.21.15.170:6800/4006535878,v1:172.21.15.170:6801/4006535878] boot 2024-05-06T13:30:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:17 smithi077 ceph-mon[22428]: osdmap e31: 5 total, 5 up, 5 in 2024-05-06T13:30:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:17 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:30:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:17 smithi077 ceph-mon[22428]: osdmap e32: 5 total, 5 up, 5 in 2024-05-06T13:30:18.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:17 smithi077 ceph-mon[27166]: purged_snaps scrub starts 2024-05-06T13:30:18.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:17 smithi077 ceph-mon[27166]: purged_snaps scrub ok 2024-05-06T13:30:18.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:17 smithi077 ceph-mon[27166]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:18.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:17 smithi077 ceph-mon[27166]: osd.4 [v2:172.21.15.170:6800/4006535878,v1:172.21.15.170:6801/4006535878] boot 2024-05-06T13:30:18.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:17 smithi077 ceph-mon[27166]: osdmap e31: 5 total, 5 up, 5 in 2024-05-06T13:30:18.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:17 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:30:18.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:17 smithi077 ceph-mon[27166]: osdmap e32: 5 total, 5 up, 5 in 2024-05-06T13:30:18.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:17 smithi170 ceph-mon[23272]: purged_snaps scrub starts 2024-05-06T13:30:18.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:17 smithi170 ceph-mon[23272]: purged_snaps scrub ok 2024-05-06T13:30:18.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:17 smithi170 ceph-mon[23272]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-05-06T13:30:18.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:17 smithi170 ceph-mon[23272]: osd.4 [v2:172.21.15.170:6800/4006535878,v1:172.21.15.170:6801/4006535878] boot 2024-05-06T13:30:18.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:17 smithi170 ceph-mon[23272]: osdmap e31: 5 total, 5 up, 5 in 2024-05-06T13:30:18.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:17 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:30:18.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:17 smithi170 ceph-mon[23272]: osdmap e32: 5 total, 5 up, 5 in 2024-05-06T13:30:18.444 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi170:vg_nvme/lv_3 2024-05-06T13:30:20.217 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:19 smithi170 ceph-mon[23272]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-05-06T13:30:20.217 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:19 smithi170 ceph-mon[23272]: osdmap e33: 5 total, 5 up, 5 in 2024-05-06T13:30:20.218 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:19 smithi170 ceph-mon[23272]: Detected new or changed devices on smithi170 2024-05-06T13:30:20.218 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:19 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:20.218 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:19 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:30:20.218 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:19 smithi170 ceph-mon[23272]: Adjusting osd_memory_target on smithi170 to 17088M 2024-05-06T13:30:20.218 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:19 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:20.218 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:19 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:20.218 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:19 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:30:20.218 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:19 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:20.218 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:19 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:30:20.218 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:19 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:20.218 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:19 smithi170 ceph-mon[23272]: osdmap e34: 5 total, 5 up, 5 in 2024-05-06T13:30:20.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:19 smithi077 ceph-mon[22428]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-05-06T13:30:20.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:19 smithi077 ceph-mon[22428]: osdmap e33: 5 total, 5 up, 5 in 2024-05-06T13:30:20.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:19 smithi077 ceph-mon[22428]: Detected new or changed devices on smithi170 2024-05-06T13:30:20.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:19 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:20.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:19 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:30:20.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:19 smithi077 ceph-mon[22428]: Adjusting osd_memory_target on smithi170 to 17088M 2024-05-06T13:30:20.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:19 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:20.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:19 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:20.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:19 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:30:20.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:19 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:20.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:19 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:30:20.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:19 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:20.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:19 smithi077 ceph-mon[22428]: osdmap e34: 5 total, 5 up, 5 in 2024-05-06T13:30:20.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:19 smithi077 ceph-mon[27166]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-05-06T13:30:20.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:19 smithi077 ceph-mon[27166]: osdmap e33: 5 total, 5 up, 5 in 2024-05-06T13:30:20.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:19 smithi077 ceph-mon[27166]: Detected new or changed devices on smithi170 2024-05-06T13:30:20.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:19 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:20.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:19 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:30:20.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:19 smithi077 ceph-mon[27166]: Adjusting osd_memory_target on smithi170 to 17088M 2024-05-06T13:30:20.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:19 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:20.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:19 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:20.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:19 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:30:20.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:19 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:20.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:19 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:30:20.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:19 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:20.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:19 smithi077 ceph-mon[27166]: osdmap e34: 5 total, 5 up, 5 in 2024-05-06T13:30:21.709 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:21 smithi170 ceph-mon[23272]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-05-06T13:30:21.709 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:21 smithi170 ceph-mon[23272]: osdmap e35: 5 total, 5 up, 5 in 2024-05-06T13:30:21.709 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:21 smithi170 ceph-mon[23272]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:30:21.709 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:21 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T13:30:21.709 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:21 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T13:30:21.709 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:21 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:21.801 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:21 smithi077 ceph-mon[22428]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-05-06T13:30:21.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:21 smithi077 ceph-mon[22428]: osdmap e35: 5 total, 5 up, 5 in 2024-05-06T13:30:21.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:21 smithi077 ceph-mon[22428]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:30:21.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:21 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T13:30:21.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:21 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T13:30:21.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:21 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:21.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:21 smithi077 ceph-mon[27166]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-05-06T13:30:21.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:21 smithi077 ceph-mon[27166]: osdmap e35: 5 total, 5 up, 5 in 2024-05-06T13:30:21.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:21 smithi077 ceph-mon[27166]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:30:21.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:21 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T13:30:21.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:21 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T13:30:21.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:21 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:23.801 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:23 smithi077 ceph-mon[22428]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-06T13:30:23.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:23 smithi077 ceph-mon[22428]: from='client.? 172.21.15.170:0/2891860138' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f0bd2da-d332-4d55-ad70-86bde46189ae"}]: dispatch 2024-05-06T13:30:23.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:23 smithi077 ceph-mon[22428]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f0bd2da-d332-4d55-ad70-86bde46189ae"}]: dispatch 2024-05-06T13:30:23.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:23 smithi077 ceph-mon[22428]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f0bd2da-d332-4d55-ad70-86bde46189ae"}]': finished 2024-05-06T13:30:23.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:23 smithi077 ceph-mon[22428]: osdmap e36: 6 total, 5 up, 6 in 2024-05-06T13:30:23.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:23 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:30:23.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:23 smithi077 ceph-mon[27166]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-06T13:30:23.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:23 smithi077 ceph-mon[27166]: from='client.? 172.21.15.170:0/2891860138' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f0bd2da-d332-4d55-ad70-86bde46189ae"}]: dispatch 2024-05-06T13:30:23.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:23 smithi077 ceph-mon[27166]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f0bd2da-d332-4d55-ad70-86bde46189ae"}]: dispatch 2024-05-06T13:30:23.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:23 smithi077 ceph-mon[27166]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f0bd2da-d332-4d55-ad70-86bde46189ae"}]': finished 2024-05-06T13:30:23.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:23 smithi077 ceph-mon[27166]: osdmap e36: 6 total, 5 up, 6 in 2024-05-06T13:30:23.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:23 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:30:23.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:23 smithi170 ceph-mon[23272]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-06T13:30:23.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:23 smithi170 ceph-mon[23272]: from='client.? 172.21.15.170:0/2891860138' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f0bd2da-d332-4d55-ad70-86bde46189ae"}]: dispatch 2024-05-06T13:30:23.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:23 smithi170 ceph-mon[23272]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f0bd2da-d332-4d55-ad70-86bde46189ae"}]: dispatch 2024-05-06T13:30:23.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:23 smithi170 ceph-mon[23272]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f0bd2da-d332-4d55-ad70-86bde46189ae"}]': finished 2024-05-06T13:30:23.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:23 smithi170 ceph-mon[23272]: osdmap e36: 6 total, 5 up, 6 in 2024-05-06T13:30:23.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:23 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:30:24.765 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:24 smithi077 ceph-mon[22428]: from='client.? 172.21.15.170:0/2324331691' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T13:30:24.766 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:24 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:30:24.766 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:24 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:30:24.766 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:24 smithi077 ceph-mon[27166]: from='client.? 172.21.15.170:0/2324331691' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T13:30:24.766 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:24 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:30:24.766 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:24 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:30:24.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:24 smithi170 ceph-mon[23272]: from='client.? 172.21.15.170:0/2324331691' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T13:30:24.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:24 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:30:24.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:24 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:30:25.801 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:25 smithi077 ceph-mon[22428]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-05-06T13:30:25.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:25 smithi077 ceph-mon[27166]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-05-06T13:30:25.802 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:25 smithi170 ceph-mon[23272]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-05-06T13:30:27.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:27 smithi170 ceph-mon[23272]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-05-06T13:30:27.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:27 smithi077 ceph-mon[22428]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-05-06T13:30:27.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:27 smithi077 ceph-mon[27166]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-05-06T13:30:29.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:29 smithi170 ceph-mon[23272]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-06T13:30:29.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:29 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-06T13:30:29.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:29 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:29.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:29 smithi170 ceph-mon[23272]: Deploying daemon osd.5 on smithi170 2024-05-06T13:30:29.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:29 smithi077 ceph-mon[22428]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-06T13:30:29.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:29 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-06T13:30:29.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:29 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:29.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:29 smithi077 ceph-mon[22428]: Deploying daemon osd.5 on smithi170 2024-05-06T13:30:29.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:29 smithi077 ceph-mon[27166]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-06T13:30:29.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:29 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-06T13:30:29.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:29 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:29.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:29 smithi077 ceph-mon[27166]: Deploying daemon osd.5 on smithi170 2024-05-06T13:30:31.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:31 smithi077 ceph-mon[22428]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-06T13:30:31.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:31 smithi077 ceph-mon[27166]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-06T13:30:31.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:31 smithi170 ceph-mon[23272]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-06T13:30:33.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:33 smithi170 ceph-mon[23272]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-06T13:30:33.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:33 smithi077 ceph-mon[22428]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-06T13:30:33.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:33 smithi077 ceph-mon[27166]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-06T13:30:34.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:34 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:30:34.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:34 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:34.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:34 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:34.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:34 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:30:34.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:34 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:30:34.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:34 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:34.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:34 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:34.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:34 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:30:34.649 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:34 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:30:34.649 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:34 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:34.649 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:34 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:34.649 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:34 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:30:36.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:35 smithi077 ceph-mon[22428]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 45 KiB/s, 0 objects/s recovering 2024-05-06T13:30:36.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:35 smithi077 ceph-mon[27166]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 45 KiB/s, 0 objects/s recovering 2024-05-06T13:30:36.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:35 smithi170 ceph-mon[23272]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 45 KiB/s, 0 objects/s recovering 2024-05-06T13:30:36.668 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 5 on host 'smithi170' 2024-05-06T13:30:37.515 DEBUG:teuthology.orchestra.run.smithi170:osd.5> sudo journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.5.service 2024-05-06T13:30:37.518 INFO:tasks.cephadm:Deploying osd.6 on smithi170 with /dev/vg_nvme/lv_2... 2024-05-06T13:30:37.518 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-06T13:30:37.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:37 smithi170 ceph-mon[23272]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-06T13:30:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:37 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:37 smithi170 ceph-mon[23272]: from='osd.5 [v2:172.21.15.170:6808/186430802,v1:172.21.15.170:6809/186430802]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T13:30:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:37 smithi170 ceph-mon[23272]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T13:30:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:37 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:38.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:37 smithi077 ceph-mon[27166]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-06T13:30:38.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:37 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:38.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:37 smithi077 ceph-mon[27166]: from='osd.5 [v2:172.21.15.170:6808/186430802,v1:172.21.15.170:6809/186430802]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T13:30:38.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:37 smithi077 ceph-mon[27166]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T13:30:38.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:37 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:38.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:37 smithi077 ceph-mon[22428]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-06T13:30:38.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:37 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:38.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:37 smithi077 ceph-mon[22428]: from='osd.5 [v2:172.21.15.170:6808/186430802,v1:172.21.15.170:6809/186430802]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T13:30:38.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:37 smithi077 ceph-mon[22428]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T13:30:38.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:37 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:39.038 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:38 smithi170 ceph-mon[23272]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-06T13:30:39.038 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:38 smithi170 ceph-mon[23272]: osdmap e37: 6 total, 5 up, 6 in 2024-05-06T13:30:39.038 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:38 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:30:39.039 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:38 smithi170 ceph-mon[23272]: from='osd.5 [v2:172.21.15.170:6808/186430802,v1:172.21.15.170:6809/186430802]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T13:30:39.039 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:38 smithi170 ceph-mon[23272]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T13:30:39.039 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 13:30:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[29596]: 2024-05-06T13:30:38.966+0000 7f2d996c3700 -1 osd.5 0 waiting for initial osdmap 2024-05-06T13:30:39.039 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 13:30:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[29596]: 2024-05-06T13:30:38.971+0000 7f2d91054700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T13:30:39.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:38 smithi077 ceph-mon[27166]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-06T13:30:39.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:38 smithi077 ceph-mon[27166]: osdmap e37: 6 total, 5 up, 6 in 2024-05-06T13:30:39.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:38 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:30:39.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:38 smithi077 ceph-mon[27166]: from='osd.5 [v2:172.21.15.170:6808/186430802,v1:172.21.15.170:6809/186430802]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T13:30:39.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:38 smithi077 ceph-mon[27166]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T13:30:39.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:38 smithi077 ceph-mon[22428]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-06T13:30:39.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:38 smithi077 ceph-mon[22428]: osdmap e37: 6 total, 5 up, 6 in 2024-05-06T13:30:39.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:38 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:30:39.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:38 smithi077 ceph-mon[22428]: from='osd.5 [v2:172.21.15.170:6808/186430802,v1:172.21.15.170:6809/186430802]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T13:30:39.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:38 smithi077 ceph-mon[22428]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T13:30:39.507 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-06T13:30:39.507 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-06T13:30:39.507 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-05-06T13:30:39.507 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-05-06T13:30:39.507 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0102218 s, 1.0 GB/s 2024-05-06T13:30:39.507 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-05-06T13:30:40.259 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:39 smithi170 ceph-mon[23272]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-06T13:30:40.260 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:39 smithi170 ceph-mon[23272]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-05-06T13:30:40.260 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:39 smithi170 ceph-mon[23272]: osdmap e38: 6 total, 5 up, 6 in 2024-05-06T13:30:40.260 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:39 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:30:40.260 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:39 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:30:40.260 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:39 smithi170 ceph-mon[23272]: osd.5 [v2:172.21.15.170:6808/186430802,v1:172.21.15.170:6809/186430802] boot 2024-05-06T13:30:40.260 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:39 smithi170 ceph-mon[23272]: osdmap e39: 6 total, 6 up, 6 in 2024-05-06T13:30:40.260 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:39 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:30:40.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:39 smithi077 ceph-mon[22428]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-06T13:30:40.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:39 smithi077 ceph-mon[22428]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-05-06T13:30:40.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:39 smithi077 ceph-mon[22428]: osdmap e38: 6 total, 5 up, 6 in 2024-05-06T13:30:40.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:39 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:30:40.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:39 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:30:40.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:39 smithi077 ceph-mon[22428]: osd.5 [v2:172.21.15.170:6808/186430802,v1:172.21.15.170:6809/186430802] boot 2024-05-06T13:30:40.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:39 smithi077 ceph-mon[22428]: osdmap e39: 6 total, 6 up, 6 in 2024-05-06T13:30:40.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:39 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:30:40.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:39 smithi077 ceph-mon[27166]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-05-06T13:30:40.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:39 smithi077 ceph-mon[27166]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-05-06T13:30:40.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:39 smithi077 ceph-mon[27166]: osdmap e38: 6 total, 5 up, 6 in 2024-05-06T13:30:40.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:39 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:30:40.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:39 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:30:40.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:39 smithi077 ceph-mon[27166]: osd.5 [v2:172.21.15.170:6808/186430802,v1:172.21.15.170:6809/186430802] boot 2024-05-06T13:30:40.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:39 smithi077 ceph-mon[27166]: osdmap e39: 6 total, 6 up, 6 in 2024-05-06T13:30:40.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:39 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:30:40.575 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi170:vg_nvme/lv_2 2024-05-06T13:30:41.235 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:40 smithi170 ceph-mon[23272]: purged_snaps scrub starts 2024-05-06T13:30:41.235 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:40 smithi170 ceph-mon[23272]: purged_snaps scrub ok 2024-05-06T13:30:41.235 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:40 smithi170 ceph-mon[23272]: osdmap e40: 6 total, 6 up, 6 in 2024-05-06T13:30:41.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:40 smithi077 ceph-mon[22428]: purged_snaps scrub starts 2024-05-06T13:30:41.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:40 smithi077 ceph-mon[22428]: purged_snaps scrub ok 2024-05-06T13:30:41.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:40 smithi077 ceph-mon[22428]: osdmap e40: 6 total, 6 up, 6 in 2024-05-06T13:30:41.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:40 smithi077 ceph-mon[27166]: purged_snaps scrub starts 2024-05-06T13:30:41.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:40 smithi077 ceph-mon[27166]: purged_snaps scrub ok 2024-05-06T13:30:41.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:40 smithi077 ceph-mon[27166]: osdmap e40: 6 total, 6 up, 6 in 2024-05-06T13:30:42.130 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:41 smithi170 ceph-mon[23272]: pgmap v127: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-06T13:30:42.130 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:41 smithi170 ceph-mon[23272]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-05-06T13:30:42.130 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:41 smithi170 ceph-mon[23272]: osdmap e41: 6 total, 6 up, 6 in 2024-05-06T13:30:42.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:41 smithi077 ceph-mon[22428]: pgmap v127: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-06T13:30:42.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:41 smithi077 ceph-mon[22428]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-05-06T13:30:42.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:41 smithi077 ceph-mon[22428]: osdmap e41: 6 total, 6 up, 6 in 2024-05-06T13:30:42.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:41 smithi077 ceph-mon[27166]: pgmap v127: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-06T13:30:42.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:41 smithi077 ceph-mon[27166]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-05-06T13:30:42.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:41 smithi077 ceph-mon[27166]: osdmap e41: 6 total, 6 up, 6 in 2024-05-06T13:30:43.725 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:43 smithi170 ceph-mon[23272]: pgmap v130: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-06T13:30:43.725 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:43 smithi170 ceph-mon[23272]: osdmap e42: 6 total, 6 up, 6 in 2024-05-06T13:30:43.725 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:43 smithi170 ceph-mon[23272]: Detected new or changed devices on smithi170 2024-05-06T13:30:43.725 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:43 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:43.725 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:43 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:30:43.725 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:43 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:30:43.725 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:43 smithi170 ceph-mon[23272]: Adjusting osd_memory_target on smithi170 to 8544M 2024-05-06T13:30:43.725 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:43 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:43.725 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:43 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:43.726 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:43 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:30:43.726 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:43 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:43.726 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:43 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:30:43.726 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:43 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:43.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:43 smithi077 ceph-mon[27166]: pgmap v130: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-06T13:30:43.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:43 smithi077 ceph-mon[27166]: osdmap e42: 6 total, 6 up, 6 in 2024-05-06T13:30:43.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:43 smithi077 ceph-mon[27166]: Detected new or changed devices on smithi170 2024-05-06T13:30:43.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:43 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:43.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:43 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:30:43.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:43 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:30:43.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:43 smithi077 ceph-mon[27166]: Adjusting osd_memory_target on smithi170 to 8544M 2024-05-06T13:30:43.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:43 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:43.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:43 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:43.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:43 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:30:43.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:43 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:43.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:43 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:30:43.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:43 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:43.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:43 smithi077 ceph-mon[22428]: pgmap v130: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-06T13:30:43.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:43 smithi077 ceph-mon[22428]: osdmap e42: 6 total, 6 up, 6 in 2024-05-06T13:30:43.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:43 smithi077 ceph-mon[22428]: Detected new or changed devices on smithi170 2024-05-06T13:30:43.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:43 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:43.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:43 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:30:43.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:43 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:30:43.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:43 smithi077 ceph-mon[22428]: Adjusting osd_memory_target on smithi170 to 8544M 2024-05-06T13:30:43.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:43 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:43.804 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:43 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:43.804 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:43 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:30:43.804 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:43 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:43.804 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:43 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:30:43.804 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:43 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:44.677 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:44 smithi170 ceph-mon[23272]: osdmap e43: 6 total, 6 up, 6 in 2024-05-06T13:30:44.677 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:44 smithi170 ceph-mon[23272]: from='client.34247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:30:44.677 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:44 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T13:30:44.677 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:44 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T13:30:44.677 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:44 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:44.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:44 smithi077 ceph-mon[22428]: osdmap e43: 6 total, 6 up, 6 in 2024-05-06T13:30:44.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:44 smithi077 ceph-mon[22428]: from='client.34247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:30:44.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:44 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T13:30:44.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:44 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T13:30:44.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:44 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:44.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:44 smithi077 ceph-mon[27166]: osdmap e43: 6 total, 6 up, 6 in 2024-05-06T13:30:44.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:44 smithi077 ceph-mon[27166]: from='client.34247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:30:44.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:44 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T13:30:44.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:44 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T13:30:44.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:44 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:45.801 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:45 smithi077 ceph-mon[22428]: pgmap v133: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-06T13:30:45.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:45 smithi077 ceph-mon[27166]: pgmap v133: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-06T13:30:45.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:45 smithi170 ceph-mon[23272]: pgmap v133: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-05-06T13:30:46.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:46 smithi077 ceph-mon[22428]: from='client.? 172.21.15.170:0/1753152879' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f6caf5a-03a5-4151-a129-90f77779e3ff"}]: dispatch 2024-05-06T13:30:46.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:46 smithi077 ceph-mon[22428]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f6caf5a-03a5-4151-a129-90f77779e3ff"}]: dispatch 2024-05-06T13:30:46.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:46 smithi077 ceph-mon[22428]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f6caf5a-03a5-4151-a129-90f77779e3ff"}]': finished 2024-05-06T13:30:46.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:46 smithi077 ceph-mon[22428]: osdmap e44: 7 total, 6 up, 7 in 2024-05-06T13:30:46.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:46 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:30:46.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:46 smithi077 ceph-mon[27166]: from='client.? 172.21.15.170:0/1753152879' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f6caf5a-03a5-4151-a129-90f77779e3ff"}]: dispatch 2024-05-06T13:30:46.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:46 smithi077 ceph-mon[27166]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f6caf5a-03a5-4151-a129-90f77779e3ff"}]: dispatch 2024-05-06T13:30:46.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:46 smithi077 ceph-mon[27166]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f6caf5a-03a5-4151-a129-90f77779e3ff"}]': finished 2024-05-06T13:30:46.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:46 smithi077 ceph-mon[27166]: osdmap e44: 7 total, 6 up, 7 in 2024-05-06T13:30:46.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:46 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:30:46.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:46 smithi170 ceph-mon[23272]: from='client.? 172.21.15.170:0/1753152879' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f6caf5a-03a5-4151-a129-90f77779e3ff"}]: dispatch 2024-05-06T13:30:46.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:46 smithi170 ceph-mon[23272]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f6caf5a-03a5-4151-a129-90f77779e3ff"}]: dispatch 2024-05-06T13:30:46.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:46 smithi170 ceph-mon[23272]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f6caf5a-03a5-4151-a129-90f77779e3ff"}]': finished 2024-05-06T13:30:46.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:46 smithi170 ceph-mon[23272]: osdmap e44: 7 total, 6 up, 7 in 2024-05-06T13:30:46.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:46 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:30:47.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:47 smithi077 ceph-mon[22428]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-05-06T13:30:47.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:47 smithi077 ceph-mon[22428]: from='client.? 172.21.15.170:0/4230136719' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T13:30:47.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:47 smithi077 ceph-mon[22428]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-05-06T13:30:47.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:47 smithi077 ceph-mon[22428]: Cluster is now healthy 2024-05-06T13:30:47.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:47 smithi077 ceph-mon[27166]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-05-06T13:30:47.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:47 smithi077 ceph-mon[27166]: from='client.? 172.21.15.170:0/4230136719' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T13:30:47.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:47 smithi077 ceph-mon[27166]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-05-06T13:30:47.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:47 smithi077 ceph-mon[27166]: Cluster is now healthy 2024-05-06T13:30:47.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:47 smithi170 ceph-mon[23272]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-05-06T13:30:47.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:47 smithi170 ceph-mon[23272]: from='client.? 172.21.15.170:0/4230136719' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T13:30:47.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:47 smithi170 ceph-mon[23272]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-05-06T13:30:47.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:47 smithi170 ceph-mon[23272]: Cluster is now healthy 2024-05-06T13:30:49.762 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:49 smithi170 ceph-mon[23272]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-06T13:30:49.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:49 smithi077 ceph-mon[22428]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-06T13:30:49.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:49 smithi077 ceph-mon[27166]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-06T13:30:51.707 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:51 smithi170 ceph-mon[23272]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-06T13:30:51.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:51 smithi077 ceph-mon[22428]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-06T13:30:51.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:51 smithi077 ceph-mon[27166]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-06T13:30:52.778 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:52 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-06T13:30:52.778 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:52 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:52.778 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:52 smithi170 ceph-mon[23272]: Deploying daemon osd.6 on smithi170 2024-05-06T13:30:52.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:52 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-06T13:30:52.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:52 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:52.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:52 smithi077 ceph-mon[22428]: Deploying daemon osd.6 on smithi170 2024-05-06T13:30:52.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:52 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-06T13:30:52.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:52 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:52.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:52 smithi077 ceph-mon[27166]: Deploying daemon osd.6 on smithi170 2024-05-06T13:30:53.668 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:53 smithi170 ceph-mon[23272]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-06T13:30:53.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:53 smithi077 ceph-mon[22428]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-06T13:30:53.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:53 smithi077 ceph-mon[27166]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-06T13:30:55.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:55 smithi077 ceph-mon[22428]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-06T13:30:55.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:55 smithi077 ceph-mon[27166]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-06T13:30:55.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:55 smithi170 ceph-mon[23272]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-06T13:30:58.389 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:58 smithi170 ceph-mon[23272]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T13:30:58.389 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:58 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:58.389 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:58 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:30:58.389 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:58 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:58.389 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:30:58 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:30:58.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:58 smithi077 ceph-mon[22428]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T13:30:58.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:58 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:58.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:58 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:30:58.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:58 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:58.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:30:58 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:30:58.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:58 smithi077 ceph-mon[27166]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T13:30:58.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:58 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:30:58.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:58 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:30:58.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:58 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:30:58.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:30:58 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:30:59.974 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 6 on host 'smithi170' 2024-05-06T13:31:00.277 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:00 smithi170 ceph-mon[23272]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T13:31:00.277 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:00 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:00.524 DEBUG:teuthology.orchestra.run.smithi170:osd.6> sudo journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.6.service 2024-05-06T13:31:00.527 INFO:tasks.cephadm:Deploying osd.7 on smithi170 with /dev/vg_nvme/lv_1... 2024-05-06T13:31:00.527 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-06T13:31:00.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:00 smithi077 ceph-mon[22428]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T13:31:00.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:00 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:00.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:00 smithi077 ceph-mon[27166]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T13:31:00.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:00 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:01.202 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:01 smithi170 ceph-mon[23272]: from='osd.6 [v2:172.21.15.170:6816/1050047795,v1:172.21.15.170:6817/1050047795]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-06T13:31:01.202 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:01 smithi170 ceph-mon[23272]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-06T13:31:01.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:01 smithi077 ceph-mon[22428]: from='osd.6 [v2:172.21.15.170:6816/1050047795,v1:172.21.15.170:6817/1050047795]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-06T13:31:01.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:01 smithi077 ceph-mon[22428]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-06T13:31:01.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:01 smithi077 ceph-mon[27166]: from='osd.6 [v2:172.21.15.170:6816/1050047795,v1:172.21.15.170:6817/1050047795]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-06T13:31:01.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:01 smithi077 ceph-mon[27166]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-06T13:31:02.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:02 smithi170 ceph-mon[23272]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T13:31:02.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:02 smithi170 ceph-mon[23272]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-06T13:31:02.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:02 smithi170 ceph-mon[23272]: osdmap e45: 7 total, 6 up, 7 in 2024-05-06T13:31:02.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:02 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:31:02.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:02 smithi170 ceph-mon[23272]: from='osd.6 [v2:172.21.15.170:6816/1050047795,v1:172.21.15.170:6817/1050047795]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T13:31:02.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:02 smithi170 ceph-mon[23272]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T13:31:02.327 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 13:31:02 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[32498]: 2024-05-06T13:31:02.161+0000 7f9996b44700 -1 osd.6 0 waiting for initial osdmap 2024-05-06T13:31:02.327 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 13:31:02 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[32498]: 2024-05-06T13:31:02.167+0000 7f998fcd8700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T13:31:02.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:02 smithi077 ceph-mon[22428]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T13:31:02.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:02 smithi077 ceph-mon[22428]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-06T13:31:02.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:02 smithi077 ceph-mon[22428]: osdmap e45: 7 total, 6 up, 7 in 2024-05-06T13:31:02.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:02 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:31:02.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:02 smithi077 ceph-mon[22428]: from='osd.6 [v2:172.21.15.170:6816/1050047795,v1:172.21.15.170:6817/1050047795]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T13:31:02.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:02 smithi077 ceph-mon[22428]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T13:31:02.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:02 smithi077 ceph-mon[27166]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T13:31:02.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:02 smithi077 ceph-mon[27166]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-06T13:31:02.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:02 smithi077 ceph-mon[27166]: osdmap e45: 7 total, 6 up, 7 in 2024-05-06T13:31:02.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:02 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:31:02.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:02 smithi077 ceph-mon[27166]: from='osd.6 [v2:172.21.15.170:6816/1050047795,v1:172.21.15.170:6817/1050047795]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T13:31:02.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:02 smithi077 ceph-mon[27166]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T13:31:02.646 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-06T13:31:02.647 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-06T13:31:02.647 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-05-06T13:31:02.647 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-05-06T13:31:02.647 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0100719 s, 1.0 GB/s 2024-05-06T13:31:02.647 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-05-06T13:31:03.273 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:03 smithi170 ceph-mon[23272]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-05-06T13:31:03.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:03 smithi170 ceph-mon[23272]: osdmap e46: 7 total, 6 up, 7 in 2024-05-06T13:31:03.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:03 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:31:03.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:03 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:31:03.274 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:03 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:03.275 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi170:vg_nvme/lv_1 2024-05-06T13:31:03.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:03 smithi077 ceph-mon[22428]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-05-06T13:31:03.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:03 smithi077 ceph-mon[22428]: osdmap e46: 7 total, 6 up, 7 in 2024-05-06T13:31:03.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:03 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:31:03.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:03 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:31:03.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:03 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:03.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:03 smithi077 ceph-mon[27166]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-05-06T13:31:03.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:03 smithi077 ceph-mon[27166]: osdmap e46: 7 total, 6 up, 7 in 2024-05-06T13:31:03.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:03 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:31:03.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:03 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:31:03.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:03 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:04.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:04 smithi170 ceph-mon[23272]: purged_snaps scrub starts 2024-05-06T13:31:04.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:04 smithi170 ceph-mon[23272]: purged_snaps scrub ok 2024-05-06T13:31:04.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:04 smithi170 ceph-mon[23272]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T13:31:04.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:04 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:31:04.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:04 smithi170 ceph-mon[23272]: osd.6 [v2:172.21.15.170:6816/1050047795,v1:172.21.15.170:6817/1050047795] boot 2024-05-06T13:31:04.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:04 smithi170 ceph-mon[23272]: osdmap e47: 7 total, 7 up, 7 in 2024-05-06T13:31:04.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:04 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:31:04.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:04 smithi077 ceph-mon[22428]: purged_snaps scrub starts 2024-05-06T13:31:04.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:04 smithi077 ceph-mon[22428]: purged_snaps scrub ok 2024-05-06T13:31:04.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:04 smithi077 ceph-mon[22428]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T13:31:04.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:04 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:31:04.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:04 smithi077 ceph-mon[22428]: osd.6 [v2:172.21.15.170:6816/1050047795,v1:172.21.15.170:6817/1050047795] boot 2024-05-06T13:31:04.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:04 smithi077 ceph-mon[22428]: osdmap e47: 7 total, 7 up, 7 in 2024-05-06T13:31:04.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:04 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:31:04.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:04 smithi077 ceph-mon[27166]: purged_snaps scrub starts 2024-05-06T13:31:04.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:04 smithi077 ceph-mon[27166]: purged_snaps scrub ok 2024-05-06T13:31:04.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:04 smithi077 ceph-mon[27166]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-05-06T13:31:04.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:04 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:31:04.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:04 smithi077 ceph-mon[27166]: osd.6 [v2:172.21.15.170:6816/1050047795,v1:172.21.15.170:6817/1050047795] boot 2024-05-06T13:31:04.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:04 smithi077 ceph-mon[27166]: osdmap e47: 7 total, 7 up, 7 in 2024-05-06T13:31:04.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:04 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:31:05.455 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:05 smithi170 ceph-mon[23272]: osdmap e48: 7 total, 7 up, 7 in 2024-05-06T13:31:05.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:05 smithi077 ceph-mon[27166]: osdmap e48: 7 total, 7 up, 7 in 2024-05-06T13:31:05.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:05 smithi077 ceph-mon[22428]: osdmap e48: 7 total, 7 up, 7 in 2024-05-06T13:31:06.290 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:06 smithi170 ceph-mon[23272]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-05-06T13:31:06.290 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:06 smithi170 ceph-mon[23272]: osdmap e49: 7 total, 7 up, 7 in 2024-05-06T13:31:06.290 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:06 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T13:31:06.290 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:06 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T13:31:06.290 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:06 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:31:06.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:06 smithi077 ceph-mon[22428]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-05-06T13:31:06.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:06 smithi077 ceph-mon[22428]: osdmap e49: 7 total, 7 up, 7 in 2024-05-06T13:31:06.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:06 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T13:31:06.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:06 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T13:31:06.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:06 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:31:06.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:06 smithi077 ceph-mon[27166]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-05-06T13:31:06.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:06 smithi077 ceph-mon[27166]: osdmap e49: 7 total, 7 up, 7 in 2024-05-06T13:31:06.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:06 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-06T13:31:06.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:06 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-06T13:31:06.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:06 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:31:07.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:07 smithi077 ceph-mon[22428]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:31:07.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:07 smithi077 ceph-mon[22428]: osdmap e50: 7 total, 7 up, 7 in 2024-05-06T13:31:07.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:07 smithi077 ceph-mon[27166]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:31:07.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:07 smithi077 ceph-mon[27166]: osdmap e50: 7 total, 7 up, 7 in 2024-05-06T13:31:07.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:07 smithi170 ceph-mon[23272]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:31:07.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:07 smithi170 ceph-mon[23272]: osdmap e50: 7 total, 7 up, 7 in 2024-05-06T13:31:08.235 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:08 smithi170 ceph-mon[23272]: pgmap v151: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail; 2/6 objects degraded (33.333%) 2024-05-06T13:31:08.236 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:08 smithi170 ceph-mon[23272]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-05-06T13:31:08.236 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:08 smithi170 ceph-mon[23272]: osdmap e51: 7 total, 7 up, 7 in 2024-05-06T13:31:08.236 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:08 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:08.236 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:08 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:31:08.236 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:08 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:31:08.236 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:08 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:31:08.236 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:08 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:08.236 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:08 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:08.236 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:08 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:31:08.236 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:08 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:31:08.236 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:08 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:31:08.236 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:08 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:08.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:08 smithi077 ceph-mon[22428]: pgmap v151: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail; 2/6 objects degraded (33.333%) 2024-05-06T13:31:08.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:08 smithi077 ceph-mon[22428]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-05-06T13:31:08.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:08 smithi077 ceph-mon[22428]: osdmap e51: 7 total, 7 up, 7 in 2024-05-06T13:31:08.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:08 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:08.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:08 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:31:08.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:08 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:31:08.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:08 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:31:08.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:08 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:08.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:08 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:08.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:08 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:31:08.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:08 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:31:08.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:08 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:31:08.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:08 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:08.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:08 smithi077 ceph-mon[27166]: pgmap v151: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail; 2/6 objects degraded (33.333%) 2024-05-06T13:31:08.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:08 smithi077 ceph-mon[27166]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-05-06T13:31:08.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:08 smithi077 ceph-mon[27166]: osdmap e51: 7 total, 7 up, 7 in 2024-05-06T13:31:08.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:08 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:08.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:08 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:31:08.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:08 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:31:08.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:08 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:31:08.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:08 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:08.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:08 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:08.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:08 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:31:08.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:08 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:31:08.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:08 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:31:08.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:08 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:09.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:09 smithi077 ceph-mon[22428]: Detected new or changed devices on smithi170 2024-05-06T13:31:09.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:09 smithi077 ceph-mon[22428]: Adjusting osd_memory_target on smithi170 to 5696M 2024-05-06T13:31:09.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:09 smithi077 ceph-mon[22428]: from='client.? 172.21.15.170:0/3679101342' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f0ec75a-6198-4200-b4ea-4f6421b7854b"}]: dispatch 2024-05-06T13:31:09.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:09 smithi077 ceph-mon[22428]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f0ec75a-6198-4200-b4ea-4f6421b7854b"}]: dispatch 2024-05-06T13:31:09.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:09 smithi077 ceph-mon[22428]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f0ec75a-6198-4200-b4ea-4f6421b7854b"}]': finished 2024-05-06T13:31:09.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:09 smithi077 ceph-mon[22428]: osdmap e52: 8 total, 7 up, 8 in 2024-05-06T13:31:09.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:09 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:31:09.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:09 smithi077 ceph-mon[27166]: Detected new or changed devices on smithi170 2024-05-06T13:31:09.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:09 smithi077 ceph-mon[27166]: Adjusting osd_memory_target on smithi170 to 5696M 2024-05-06T13:31:09.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:09 smithi077 ceph-mon[27166]: from='client.? 172.21.15.170:0/3679101342' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f0ec75a-6198-4200-b4ea-4f6421b7854b"}]: dispatch 2024-05-06T13:31:09.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:09 smithi077 ceph-mon[27166]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f0ec75a-6198-4200-b4ea-4f6421b7854b"}]: dispatch 2024-05-06T13:31:09.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:09 smithi077 ceph-mon[27166]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f0ec75a-6198-4200-b4ea-4f6421b7854b"}]': finished 2024-05-06T13:31:09.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:09 smithi077 ceph-mon[27166]: osdmap e52: 8 total, 7 up, 8 in 2024-05-06T13:31:09.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:09 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:31:09.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:09 smithi170 ceph-mon[23272]: Detected new or changed devices on smithi170 2024-05-06T13:31:09.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:09 smithi170 ceph-mon[23272]: Adjusting osd_memory_target on smithi170 to 5696M 2024-05-06T13:31:09.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:09 smithi170 ceph-mon[23272]: from='client.? 172.21.15.170:0/3679101342' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f0ec75a-6198-4200-b4ea-4f6421b7854b"}]: dispatch 2024-05-06T13:31:09.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:09 smithi170 ceph-mon[23272]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f0ec75a-6198-4200-b4ea-4f6421b7854b"}]: dispatch 2024-05-06T13:31:09.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:09 smithi170 ceph-mon[23272]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f0ec75a-6198-4200-b4ea-4f6421b7854b"}]': finished 2024-05-06T13:31:09.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:09 smithi170 ceph-mon[23272]: osdmap e52: 8 total, 7 up, 8 in 2024-05-06T13:31:09.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:09 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:31:10.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:10 smithi077 ceph-mon[22428]: pgmap v153: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 2/6 objects degraded (33.333%) 2024-05-06T13:31:10.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:10 smithi077 ceph-mon[22428]: from='client.? 172.21.15.170:0/830958455' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T13:31:10.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:10 smithi077 ceph-mon[27166]: pgmap v153: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 2/6 objects degraded (33.333%) 2024-05-06T13:31:10.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:10 smithi077 ceph-mon[27166]: from='client.? 172.21.15.170:0/830958455' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T13:31:10.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:10 smithi170 ceph-mon[23272]: pgmap v153: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 2/6 objects degraded (33.333%) 2024-05-06T13:31:10.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:10 smithi170 ceph-mon[23272]: from='client.? 172.21.15.170:0/830958455' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-06T13:31:11.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:11 smithi077 ceph-mon[22428]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-05-06T13:31:11.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:11 smithi077 ceph-mon[22428]: Cluster is now healthy 2024-05-06T13:31:11.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:11 smithi077 ceph-mon[27166]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-05-06T13:31:11.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:11 smithi077 ceph-mon[27166]: Cluster is now healthy 2024-05-06T13:31:11.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:11 smithi170 ceph-mon[23272]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-05-06T13:31:11.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:11 smithi170 ceph-mon[23272]: Cluster is now healthy 2024-05-06T13:31:12.486 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:12 smithi170 ceph-mon[23272]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-05-06T13:31:12.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:12 smithi077 ceph-mon[22428]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-05-06T13:31:12.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:12 smithi077 ceph-mon[27166]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-05-06T13:31:14.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:14 smithi077 ceph-mon[22428]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-05-06T13:31:14.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:14 smithi077 ceph-mon[27166]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-05-06T13:31:14.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:14 smithi170 ceph-mon[23272]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-05-06T13:31:16.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:16 smithi170 ceph-mon[23272]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-06T13:31:16.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:16 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-06T13:31:16.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:16 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:31:16.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:16 smithi077 ceph-mon[22428]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-06T13:31:16.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:16 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-06T13:31:16.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:16 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:31:16.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:16 smithi077 ceph-mon[27166]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-06T13:31:16.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:16 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-06T13:31:16.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:16 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:31:17.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:17 smithi077 ceph-mon[22428]: Deploying daemon osd.7 on smithi170 2024-05-06T13:31:17.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:17 smithi077 ceph-mon[27166]: Deploying daemon osd.7 on smithi170 2024-05-06T13:31:17.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:17 smithi170 ceph-mon[23272]: Deploying daemon osd.7 on smithi170 2024-05-06T13:31:18.490 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:18 smithi170 ceph-mon[23272]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-05-06T13:31:18.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:18 smithi077 ceph-mon[22428]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-05-06T13:31:18.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:18 smithi077 ceph-mon[27166]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-05-06T13:31:20.363 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:20 smithi170 ceph-mon[23272]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-06T13:31:20.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:20 smithi077 ceph-mon[22428]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-06T13:31:20.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:20 smithi077 ceph-mon[27166]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-06T13:31:21.749 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:21 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:21.750 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:21 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:31:21.750 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:21 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:31:21.750 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:21 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:31:21.750 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:21 smithi170 ceph-mon[23272]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 45 KiB/s, 0 objects/s recovering 2024-05-06T13:31:21.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:21 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:21.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:21 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:31:21.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:21 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:31:21.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:21 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:31:21.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:21 smithi077 ceph-mon[22428]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 45 KiB/s, 0 objects/s recovering 2024-05-06T13:31:21.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:21 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:21.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:21 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:31:21.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:21 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:31:21.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:21 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:31:21.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:21 smithi077 ceph-mon[27166]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 45 KiB/s, 0 objects/s recovering 2024-05-06T13:31:23.357 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 7 on host 'smithi170' 2024-05-06T13:31:23.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:23 smithi170 ceph-mon[23272]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-06T13:31:23.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:23 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:23.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:23 smithi077 ceph-mon[22428]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-06T13:31:23.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:23 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:23.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:23 smithi077 ceph-mon[27166]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-06T13:31:23.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:23 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:24.001 DEBUG:teuthology.orchestra.run.smithi170:osd.7> sudo journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.7.service 2024-05-06T13:31:24.003 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-05-06T13:31:24.003 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph osd stat -f json 2024-05-06T13:31:24.568 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:24 smithi077 ceph-mon[22428]: from='osd.7 [v2:172.21.15.170:6824/604573004,v1:172.21.15.170:6825/604573004]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-06T13:31:24.568 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:24 smithi077 ceph-mon[22428]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-06T13:31:24.568 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:24 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:24.568 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:24 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:31:24.568 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:24 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:31:24.569 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:24 smithi077 ceph-mon[27166]: from='osd.7 [v2:172.21.15.170:6824/604573004,v1:172.21.15.170:6825/604573004]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-06T13:31:24.569 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:24 smithi077 ceph-mon[27166]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-06T13:31:24.569 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:24 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:24.569 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:24 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:31:24.570 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:24 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:31:24.766 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:24 smithi170 ceph-mon[23272]: from='osd.7 [v2:172.21.15.170:6824/604573004,v1:172.21.15.170:6825/604573004]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-06T13:31:24.766 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:24 smithi170 ceph-mon[23272]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-06T13:31:24.766 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:24 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:24.766 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:24 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:31:24.767 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:24 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:31:25.590 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:25 smithi077 ceph-mon[22428]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-06T13:31:25.590 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:25 smithi077 ceph-mon[22428]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-06T13:31:25.590 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:25 smithi077 ceph-mon[22428]: osdmap e53: 8 total, 7 up, 8 in 2024-05-06T13:31:25.590 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:25 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:31:25.591 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:25 smithi077 ceph-mon[22428]: from='osd.7 [v2:172.21.15.170:6824/604573004,v1:172.21.15.170:6825/604573004]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T13:31:25.591 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:25 smithi077 ceph-mon[22428]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T13:31:25.591 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:25 smithi077 ceph-mon[27166]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-06T13:31:25.591 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:25 smithi077 ceph-mon[27166]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-06T13:31:25.591 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:25 smithi077 ceph-mon[27166]: osdmap e53: 8 total, 7 up, 8 in 2024-05-06T13:31:25.591 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:25 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:31:25.591 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:25 smithi077 ceph-mon[27166]: from='osd.7 [v2:172.21.15.170:6824/604573004,v1:172.21.15.170:6825/604573004]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T13:31:25.591 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:25 smithi077 ceph-mon[27166]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T13:31:25.799 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:25 smithi170 ceph-mon[23272]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-06T13:31:25.800 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:25 smithi170 ceph-mon[23272]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-06T13:31:25.800 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:25 smithi170 ceph-mon[23272]: osdmap e53: 8 total, 7 up, 8 in 2024-05-06T13:31:25.800 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:25 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:31:25.800 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:25 smithi170 ceph-mon[23272]: from='osd.7 [v2:172.21.15.170:6824/604573004,v1:172.21.15.170:6825/604573004]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T13:31:25.800 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:25 smithi170 ceph-mon[23272]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T13:31:25.800 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 13:31:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[35425]: 2024-05-06T13:31:25.466+0000 7f9733359700 -1 osd.7 0 waiting for initial osdmap 2024-05-06T13:31:25.800 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 13:31:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[35425]: 2024-05-06T13:31:25.473+0000 7f972d4ef700 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T13:31:25.992 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:31:26.619 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:26 smithi077 ceph-mon[27166]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-05-06T13:31:26.619 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:26 smithi077 ceph-mon[27166]: osdmap e54: 8 total, 7 up, 8 in 2024-05-06T13:31:26.619 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:26 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:31:26.619 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:26 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:31:26.619 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:26 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/341880467' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-06T13:31:26.620 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:26 smithi077 ceph-mon[22428]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-05-06T13:31:26.620 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:26 smithi077 ceph-mon[22428]: osdmap e54: 8 total, 7 up, 8 in 2024-05-06T13:31:26.620 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:26 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:31:26.620 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:26 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:31:26.620 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:26 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/341880467' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-06T13:31:26.621 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":54,"num_osds":8,"num_up_osds":7,"osd_up_since":1715002263,"num_in_osds":8,"osd_in_since":1715002268,"num_remapped_pgs":0} 2024-05-06T13:31:26.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:26 smithi170 ceph-mon[23272]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-05-06T13:31:26.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:26 smithi170 ceph-mon[23272]: osdmap e54: 8 total, 7 up, 8 in 2024-05-06T13:31:26.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:26 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:31:26.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:26 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:31:26.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:26 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/341880467' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-06T13:31:27.622 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph osd stat -f json 2024-05-06T13:31:27.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[22428]: purged_snaps scrub starts 2024-05-06T13:31:27.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[22428]: purged_snaps scrub ok 2024-05-06T13:31:27.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[22428]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-06T13:31:27.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[22428]: osd.7 [v2:172.21.15.170:6824/604573004,v1:172.21.15.170:6825/604573004] boot 2024-05-06T13:31:27.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[22428]: osdmap e55: 8 total, 8 up, 8 in 2024-05-06T13:31:27.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:31:27.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:27.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:31:27.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:31:27.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:31:27.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:31:27.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:27.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:27.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:31:27.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:31:27.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:31:27.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:27.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[27166]: purged_snaps scrub starts 2024-05-06T13:31:27.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[27166]: purged_snaps scrub ok 2024-05-06T13:31:27.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[27166]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-06T13:31:27.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[27166]: osd.7 [v2:172.21.15.170:6824/604573004,v1:172.21.15.170:6825/604573004] boot 2024-05-06T13:31:27.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[27166]: osdmap e55: 8 total, 8 up, 8 in 2024-05-06T13:31:27.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:31:27.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:27.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:31:27.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:31:27.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:31:27.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:31:27.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:27.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:27.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:31:27.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:31:27.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:31:27.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:27 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:27.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:27 smithi170 ceph-mon[23272]: purged_snaps scrub starts 2024-05-06T13:31:27.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:27 smithi170 ceph-mon[23272]: purged_snaps scrub ok 2024-05-06T13:31:27.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:27 smithi170 ceph-mon[23272]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-05-06T13:31:27.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:27 smithi170 ceph-mon[23272]: osd.7 [v2:172.21.15.170:6824/604573004,v1:172.21.15.170:6825/604573004] boot 2024-05-06T13:31:27.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:27 smithi170 ceph-mon[23272]: osdmap e55: 8 total, 8 up, 8 in 2024-05-06T13:31:27.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:27 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:31:27.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:27 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:27.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:27 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:31:27.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:27 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:31:27.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:27 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:31:27.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:27 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:31:27.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:27 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:27.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:27 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:27.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:27 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:31:27.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:27 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:31:27.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:27 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:31:27.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:27 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:28.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:28 smithi077 ceph-mon[27166]: Detected new or changed devices on smithi170 2024-05-06T13:31:28.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:28 smithi077 ceph-mon[27166]: Adjusting osd_memory_target on smithi170 to 4272M 2024-05-06T13:31:28.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:28 smithi077 ceph-mon[27166]: osdmap e56: 8 total, 8 up, 8 in 2024-05-06T13:31:28.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:28 smithi077 ceph-mon[22428]: Detected new or changed devices on smithi170 2024-05-06T13:31:28.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:28 smithi077 ceph-mon[22428]: Adjusting osd_memory_target on smithi170 to 4272M 2024-05-06T13:31:28.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:28 smithi077 ceph-mon[22428]: osdmap e56: 8 total, 8 up, 8 in 2024-05-06T13:31:28.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:28 smithi170 ceph-mon[23272]: Detected new or changed devices on smithi170 2024-05-06T13:31:28.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:28 smithi170 ceph-mon[23272]: Adjusting osd_memory_target on smithi170 to 4272M 2024-05-06T13:31:28.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:28 smithi170 ceph-mon[23272]: osdmap e56: 8 total, 8 up, 8 in 2024-05-06T13:31:29.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:29 smithi077 ceph-mon[22428]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:29.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:29 smithi077 ceph-mon[22428]: osdmap e57: 8 total, 8 up, 8 in 2024-05-06T13:31:29.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:29 smithi077 ceph-mon[22428]: osdmap e58: 8 total, 8 up, 8 in 2024-05-06T13:31:29.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:29 smithi077 ceph-mon[27166]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:29.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:29 smithi077 ceph-mon[27166]: osdmap e57: 8 total, 8 up, 8 in 2024-05-06T13:31:29.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:29 smithi077 ceph-mon[27166]: osdmap e58: 8 total, 8 up, 8 in 2024-05-06T13:31:29.652 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:31:29.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:29 smithi170 ceph-mon[23272]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:29.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:29 smithi170 ceph-mon[23272]: osdmap e57: 8 total, 8 up, 8 in 2024-05-06T13:31:29.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:29 smithi170 ceph-mon[23272]: osdmap e58: 8 total, 8 up, 8 in 2024-05-06T13:31:30.280 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":58,"num_osds":8,"num_up_osds":8,"osd_up_since":1715002286,"num_in_osds":8,"osd_in_since":1715002268,"num_remapped_pgs":0} 2024-05-06T13:31:30.280 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-06T13:31:30.492 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T13:31:30.741 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:30 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/301024875' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-06T13:31:30.759 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:30 smithi077 ceph-mon[22428]: osdmap e59: 8 total, 8 up, 8 in 2024-05-06T13:31:30.759 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:30 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/301024875' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-06T13:31:30.759 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:30 smithi077 ceph-mon[27166]: osdmap e59: 8 total, 8 up, 8 in 2024-05-06T13:31:30.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:30 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/301024875' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-06T13:31:30.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:30 smithi170 ceph-mon[23272]: osdmap e59: 8 total, 8 up, 8 in 2024-05-06T13:31:31.527 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:31 smithi077 ceph-mon[22428]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:31.527 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:31 smithi077 ceph-mon[27166]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:31.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:31 smithi170 ceph-mon[23272]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:32.150 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:31:32.150 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":59,"fsid":"20e59cb4-0bac-11ef-bc97-c7b262605968","created":"2024-05-06T13:26:37.391183+0000","modified":"2024-05-06T13:31:30.479889+0000","last_up_change":"2024-05-06T13:31:26.464116+0000","last_in_change":"2024-05-06T13:31:08.958161+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-06T13:29:32.660551+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"8baf11f2-252b-4937-8684-28c621b69971","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":834793539},{"type":"v1","addr":"172.21.15.77:6803","nonce":834793539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":834793539},{"type":"v1","addr":"172.21.15.77:6805","nonce":834793539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":834793539},{"type":"v1","addr":"172.21.15.77:6809","nonce":834793539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":834793539},{"type":"v1","addr":"172.21.15.77:6807","nonce":834793539}]},"public_addr":"172.21.15.77:6803/834793539","cluster_addr":"172.21.15.77:6805/834793539","heartbeat_back_addr":"172.21.15.77:6809/834793539","heartbeat_front_addr":"172.21.15.77:6807/834793539","state":["exists","up"]},{"osd":1,"uuid":"69b4ced2-2786-41d0-be2d-2acfcb52ece1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2133267326},{"type":"v1","addr":"172.21.15.77:6811","nonce":2133267326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2133267326},{"type":"v1","addr":"172.21.15.77:6813","nonce":2133267326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":2133267326},{"type":"v1","addr":"172.21.15.77:6817","nonce":2133267326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2133267326},{"type":"v1","addr":"172.21.15.77:6815","nonce":2133267326}]},"public_addr":"172.21.15.77:6811/2133267326","cluster_addr":"172.21.15.77:6813/2133267326","heartbeat_back_addr":"172.21.15.77:6817/2133267326","heartbeat_front_addr":"172.21.15.77:6815/2133267326","state":["exists","up"]},{"osd":2,"uuid":"dcab9e69-bf02-4b9f-9ff1-5dbb5d71a557","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":2522642178},{"type":"v1","addr":"172.21.15.77:6819","nonce":2522642178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":2522642178},{"type":"v1","addr":"172.21.15.77:6821","nonce":2522642178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2522642178},{"type":"v1","addr":"172.21.15.77:6825","nonce":2522642178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":2522642178},{"type":"v1","addr":"172.21.15.77:6823","nonce":2522642178}]},"public_addr":"172.21.15.77:6819/2522642178","cluster_addr":"172.21.15.77:6821/2522642178","heartbeat_back_addr":"172.21.15.77:6825/2522642178","heartbeat_front_addr":"172.21.15.77:6823/2522642178","state":["exists","up"]},{"osd":3,"uuid":"4b177913-e714-4f93-99b4-e2da20777d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3252753833},{"type":"v1","addr":"172.21.15.77:6827","nonce":3252753833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3252753833},{"type":"v1","addr":"172.21.15.77:6829","nonce":3252753833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3252753833},{"type":"v1","addr":"172.21.15.77:6833","nonce":3252753833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3252753833},{"type":"v1","addr":"172.21.15.77:6831","nonce":3252753833}]},"public_addr":"172.21.15.77:6827/3252753833","cluster_addr":"172.21.15.77:6829/3252753833","heartbeat_back_addr":"172.21.15.77:6833/3252753833","heartbeat_front_addr":"172.21.15.77:6831/3252753833","state":["exists","up"]},{"osd":4,"uuid":"ce6d16fa-5c8c-4a94-852b-f8193d47d25f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":4006535878},{"type":"v1","addr":"172.21.15.170:6801","nonce":4006535878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":4006535878},{"type":"v1","addr":"172.21.15.170:6803","nonce":4006535878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":4006535878},{"type":"v1","addr":"172.21.15.170:6807","nonce":4006535878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":4006535878},{"type":"v1","addr":"172.21.15.170:6805","nonce":4006535878}]},"public_addr":"172.21.15.170:6801/4006535878","cluster_addr":"172.21.15.170:6803/4006535878","heartbeat_back_addr":"172.21.15.170:6807/4006535878","heartbeat_front_addr":"172.21.15.170:6805/4006535878","state":["exists","up"]},{"osd":5,"uuid":"4f0bd2da-d332-4d55-ad70-86bde46189ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":186430802},{"type":"v1","addr":"172.21.15.170:6809","nonce":186430802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":186430802},{"type":"v1","addr":"172.21.15.170:6811","nonce":186430802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":186430802},{"type":"v1","addr":"172.21.15.170:6815","nonce":186430802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":186430802},{"type":"v1","addr":"172.21.15.170:6813","nonce":186430802}]},"public_addr":"172.21.15.170:6809/186430802","cluster_addr":"172.21.15.170:6811/186430802","heartbeat_back_addr":"172.21.15.170:6815/186430802","heartbeat_front_addr":"172.21.15.170:6813/186430802","state":["exists","up"]},{"osd":6,"uuid":"2f6caf5a-03a5-4151-a129-90f77779e3ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1050047795},{"type":"v1","addr":"172.21.15.170:6817","nonce":1050047795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1050047795},{"type":"v1","addr":"172.21.15.170:6819","nonce":1050047795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1050047795},{"type":"v1","addr":"172.21.15.170:6823","nonce":1050047795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1050047795},{"type":"v1","addr":"172.21.15.170:6821","nonce":1050047795}]},"public_addr":"172.21.15.170:6817/1050047795","cluster_addr":"172.21.15.170:6819/1050047795","heartbeat_back_addr":"172.21.15.170:6823/1050047795","heartbeat_front_addr":"172.21.15.170:6821/1050047795","state":["exists","up"]},{"osd":7,"uuid":"4f0ec75a-6198-4200-b4ea-4f6421b7854b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":604573004},{"type":"v1","addr":"172.21.15.170:6825","nonce":604573004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":604573004},{"type":"v1","addr":"172.21.15.170:6827","nonce":604573004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":604573004},{"type":"v1","addr":"172.21.15.170:6831","nonce":604573004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":604573004},{"type":"v1","addr":"172.21.15.170:6829","nonce":604573004}]},"public_addr":"172.21.15.170:6825/604573004","cluster_addr":"172.21.15.170:6827/604573004","heartbeat_back_addr":"172.21.15.170:6831/604573004","heartbeat_front_addr":"172.21.15.170:6829/604573004","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T13:28:42.270920+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T13:29:05.303372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T13:29:29.095238+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T13:29:53.475953+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T13:30:15.076693+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T13:30:38.107348+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T13:31:01.288168+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T13:31:24.484479+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:6801/1922446457":"2024-05-07T13:27:24.299634+0000","172.21.15.77:0/554686488":"2024-05-07T13:27:24.299634+0000","172.21.15.77:0/3193136642":"2024-05-07T13:27:02.199372+0000","172.21.15.77:0/2039164702":"2024-05-07T13:27:02.199372+0000","172.21.15.77:0/1112825224":"2024-05-07T13:27:24.299634+0000","172.21.15.77:6800/2007005358":"2024-05-07T13:27:02.199372+0000","172.21.15.77:6800/1922446457":"2024-05-07T13:27:24.299634+0000","172.21.15.77:0/3698539339":"2024-05-07T13:27:24.299634+0000","172.21.15.77:6801/2007005358":"2024-05-07T13:27:02.199372+0000","172.21.15.77:0/461831909":"2024-05-07T13:27:02.199372+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-06T13:31:32.784 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:32 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/221541511' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T13:31:32.785 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:32 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/221541511' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T13:31:32.807 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-06T13:29:32.660551+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-05-06T13:31:32.808 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-05-06T13:31:32.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:32 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/221541511' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T13:31:33.013 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T13:31:33.777 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:33 smithi077 ceph-mon[22428]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:33.777 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:33 smithi077 ceph-mon[27166]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:33.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:33 smithi170 ceph-mon[23272]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:34.799 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-05-06T13:31:35.460 INFO:tasks.cephadm:Adding prometheus.a on smithi170 2024-05-06T13:31:35.460 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph orch apply prometheus '1;smithi170=a' 2024-05-06T13:31:35.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:35 smithi077 ceph-mon[22428]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-06T13:31:35.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:35 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3124846855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-06T13:31:35.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:35 smithi077 ceph-mon[27166]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-06T13:31:35.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:35 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3124846855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-06T13:31:35.774 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:35 smithi170 ceph-mon[23272]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-05-06T13:31:35.775 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:35 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/3124846855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-06T13:31:37.369 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled prometheus update... 2024-05-06T13:31:37.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:37 smithi077 ceph-mon[22428]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-06T13:31:37.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:37 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:37.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:37 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:31:37.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:37 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:31:37.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:37 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:31:37.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:37 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:37.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:37 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-05-06T13:31:37.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:37 smithi077 ceph-mon[27166]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-06T13:31:37.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:37 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:37.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:37 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:31:37.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:37 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:31:37.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:37 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:31:37.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:37 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:37.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:37 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-05-06T13:31:37.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:37 smithi170 ceph-mon[23272]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-06T13:31:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:37 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:37 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:31:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:37 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:31:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:37 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:31:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:37 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' 2024-05-06T13:31:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:37 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-05-06T13:31:38.147 DEBUG:teuthology.orchestra.run.smithi170:prometheus.a> sudo journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@prometheus.a.service 2024-05-06T13:31:38.150 INFO:tasks.cephadm:Adding node-exporter.a on smithi077 2024-05-06T13:31:38.150 INFO:tasks.cephadm:Adding node-exporter.b on smithi170 2024-05-06T13:31:38.150 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph orch apply node-exporter '2;smithi077=a;smithi170=b' 2024-05-06T13:31:38.702 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:38 smithi077 ceph-mon[22428]: from='client.34313 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi170=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:31:38.702 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:38 smithi077 ceph-mon[22428]: Saving service prometheus spec with placement smithi170=a;count:1 2024-05-06T13:31:38.703 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:38 smithi077 ceph-mon[22428]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-05-06T13:31:38.703 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:38 smithi077 ceph-mon[22428]: mgrmap e16: y(active, since 4m), standbys: x 2024-05-06T13:31:38.703 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:38 smithi077 ceph-mon[27166]: from='client.34313 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi170=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:31:38.703 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:38 smithi077 ceph-mon[27166]: Saving service prometheus spec with placement smithi170=a;count:1 2024-05-06T13:31:38.703 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:38 smithi077 ceph-mon[27166]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-05-06T13:31:38.703 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:38 smithi077 ceph-mon[27166]: mgrmap e16: y(active, since 4m), standbys: x 2024-05-06T13:31:38.703 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:38 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ignoring --setuser ceph since I am not root 2024-05-06T13:31:38.703 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:38 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ignoring --setgroup ceph since I am not root 2024-05-06T13:31:38.703 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:38 smithi077 ceph-mgr[22664]: -- 172.21.15.77:0/2844993588 <== mon.1 v2:172.21.15.77:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559883476340 con 0x55988344e800 2024-05-06T13:31:38.704 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:38 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:31:38.576+0000 7f63a0217000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T13:31:38.704 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:38 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:31:38.702+0000 7f63a0217000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:31:38.704 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:38 smithi170 ceph-mon[23272]: from='client.34313 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi170=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:31:38.704 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:38 smithi170 ceph-mon[23272]: Saving service prometheus spec with placement smithi170=a;count:1 2024-05-06T13:31:38.704 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:38 smithi170 ceph-mon[23272]: from='mgr.14150 172.21.15.77:0/2915885161' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-05-06T13:31:38.704 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:38 smithi170 ceph-mon[23272]: mgrmap e16: y(active, since 4m), standbys: x 2024-05-06T13:31:38.704 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ignoring --setuser ceph since I am not root 2024-05-06T13:31:38.704 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ignoring --setgroup ceph since I am not root 2024-05-06T13:31:38.705 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:38 smithi170 ceph-mgr[24453]: -- 172.21.15.170:0/3197996341 <== mon.2 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563a521c4340 con 0x563a5219cc00 2024-05-06T13:31:38.705 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:31:38.567+0000 7faa8ea73000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T13:31:38.705 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:31:38.703+0000 7faa8ea73000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:31:39.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:38 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:31:38.799+0000 7f63a0217000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T13:31:39.075 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:31:38.806+0000 7faa8ea73000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T13:31:39.378 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:39 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:31:39.115+0000 7faa8ea73000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T13:31:39.379 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:39 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:31:39.199+0000 7faa8ea73000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T13:31:39.551 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:39 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:31:39.100+0000 7f63a0217000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T13:31:39.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:39 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:31:39.181+0000 7f63a0217000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T13:31:40.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:40 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:31:40.284+0000 7f63a0217000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T13:31:40.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:40 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:31:40.446+0000 7f63a0217000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T13:31:40.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:40 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:31:40.527+0000 7f63a0217000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T13:31:40.826 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:40 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:31:40.342+0000 7faa8ea73000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T13:31:40.826 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:40 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:31:40.503+0000 7faa8ea73000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T13:31:40.826 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:40 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:31:40.583+0000 7faa8ea73000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T13:31:41.257 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:40 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:31:40.837+0000 7f63a0217000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T13:31:41.257 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:40 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:31:40.927+0000 7f63a0217000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T13:31:41.257 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:31:41.019+0000 7f63a0217000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T13:31:41.257 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:31:41.257+0000 7f63a0217000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:31:41.317 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:40 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:31:40.894+0000 7faa8ea73000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T13:31:41.317 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:40 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:31:40.984+0000 7faa8ea73000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T13:31:41.317 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:41 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:31:41.076+0000 7faa8ea73000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T13:31:41.317 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:41 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:31:41.316+0000 7faa8ea73000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:31:42.051 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:31:41.687+0000 7f63a0217000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T13:31:42.076 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:41 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:31:41.753+0000 7faa8ea73000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T13:31:42.548 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:31:42.166+0000 7f63a0217000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T13:31:42.548 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:31:42.258+0000 7f63a0217000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T13:31:42.548 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:31:42.547+0000 7f63a0217000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T13:31:42.575 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:42 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:31:42.239+0000 7faa8ea73000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T13:31:42.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:42 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:31:42.332+0000 7faa8ea73000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T13:31:42.802 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:31:42.637+0000 7f63a0217000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T13:31:43.076 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:42 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:31:42.623+0000 7faa8ea73000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T13:31:43.076 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:42 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:31:42.711+0000 7faa8ea73000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T13:31:43.302 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:31:43.035+0000 7f63a0217000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T13:31:43.302 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:31:43.166+0000 7f63a0217000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T13:31:43.302 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:31:43.260+0000 7f63a0217000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T13:31:43.447 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:43 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:31:43.111+0000 7faa8ea73000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T13:31:43.448 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:43 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:31:43.243+0000 7faa8ea73000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T13:31:43.448 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:43 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:31:43.335+0000 7faa8ea73000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T13:31:43.801 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:31:43.374+0000 7f63a0217000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T13:31:43.802 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:31:43.463+0000 7f63a0217000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T13:31:43.825 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:43 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:31:43.447+0000 7faa8ea73000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T13:31:43.826 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:43 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:31:43.536+0000 7faa8ea73000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T13:31:44.302 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:44 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:31:44.004+0000 7f63a0217000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T13:31:44.418 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:44 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:31:44.079+0000 7faa8ea73000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T13:31:44.613 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:44 smithi077 ceph-mon[22428]: Active manager daemon y restarted 2024-05-06T13:31:44.613 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:44 smithi077 ceph-mon[22428]: Activating manager daemon y 2024-05-06T13:31:44.613 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:44 smithi077 ceph-mon[22428]: osdmap e60: 8 total, 8 up, 8 in 2024-05-06T13:31:44.614 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:44 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:31:44.357+0000 7f63a0217000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T13:31:44.614 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:44 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: [06/May/2024:13:31:44] ENGINE Bus STARTING 2024-05-06T13:31:44.614 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:44 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: CherryPy Checker: 2024-05-06T13:31:44.614 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:44 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: The Application mounted at '' has an empty config. 2024-05-06T13:31:44.614 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:44 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:31:44.615 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:44 smithi077 ceph-mon[27166]: Active manager daemon y restarted 2024-05-06T13:31:44.615 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:44 smithi077 ceph-mon[27166]: Activating manager daemon y 2024-05-06T13:31:44.615 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:44 smithi077 ceph-mon[27166]: osdmap e60: 8 total, 8 up, 8 in 2024-05-06T13:31:44.708 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:44 smithi170 ceph-mon[23272]: Active manager daemon y restarted 2024-05-06T13:31:44.709 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:44 smithi170 ceph-mon[23272]: Activating manager daemon y 2024-05-06T13:31:44.709 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:44 smithi170 ceph-mon[23272]: osdmap e60: 8 total, 8 up, 8 in 2024-05-06T13:31:44.709 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:44 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:31:44.446+0000 7faa8ea73000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T13:31:44.709 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:44 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: [06/May/2024:13:31:44] ENGINE Bus STARTING 2024-05-06T13:31:44.709 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:44 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: CherryPy Checker: 2024-05-06T13:31:44.709 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:44 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: The Application mounted at '' has an empty config. 2024-05-06T13:31:44.709 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:44 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: 2024-05-06T13:31:44.709 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:44 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: [06/May/2024:13:31:44] ENGINE Serving on http://:::9283 2024-05-06T13:31:44.709 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:31:44 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: [06/May/2024:13:31:44] ENGINE Bus STARTED 2024-05-06T13:31:44.906 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:44 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: [06/May/2024:13:31:44] ENGINE Serving on http://:::9283 2024-05-06T13:31:44.906 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:44 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: [06/May/2024:13:31:44] ENGINE Bus STARTED 2024-05-06T13:31:44.906 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:44 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: [06/May/2024:13:31:44] ENGINE Bus STARTING 2024-05-06T13:31:45.208 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:45 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: [06/May/2024:13:31:45] ENGINE Serving on https://172.21.15.77:7150 2024-05-06T13:31:45.208 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:31:45 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: [06/May/2024:13:31:45] ENGINE Bus STARTED 2024-05-06T13:31:45.443 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled node-exporter update... 2024-05-06T13:31:45.456 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: mgrmap e17: y(active, starting, since 0.0567234s), standbys: x 2024-05-06T13:31:45.456 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:31:45.456 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:31:45.456 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:31:45.456 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T13:31:45.457 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T13:31:45.457 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:31:45.457 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:31:45.457 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:31:45.457 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:31:45.457 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:31:45.457 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:31:45.457 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:31:45.457 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:31:45.457 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T13:31:45.457 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T13:31:45.457 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T13:31:45.457 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: Manager daemon y is now available 2024-05-06T13:31:45.457 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:31:45.458 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:31:45.458 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: Standby manager daemon x restarted 2024-05-06T13:31:45.458 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: Standby manager daemon x started 2024-05-06T13:31:45.458 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:31:45.458 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:31:45.458 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.170:0/661819691' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T13:31:45.458 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.170:0/661819691' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:31:45.458 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.170:0/661819691' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T13:31:45.458 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.170:0/661819691' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:31:45.458 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:45.458 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:31:45.458 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:31:45.458 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:45 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:45.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: mgrmap e17: y(active, starting, since 0.0567234s), standbys: x 2024-05-06T13:31:45.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:31:45.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:31:45.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:31:45.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T13:31:45.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T13:31:45.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:31:45.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:31:45.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:31:45.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:31:45.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:31:45.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:31:45.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:31:45.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:31:45.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T13:31:45.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T13:31:45.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T13:31:45.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: Manager daemon y is now available 2024-05-06T13:31:45.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:31:45.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:31:45.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: Standby manager daemon x restarted 2024-05-06T13:31:45.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: Standby manager daemon x started 2024-05-06T13:31:45.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:31:45.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:31:45.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.170:0/661819691' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T13:31:45.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.170:0/661819691' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:31:45.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.170:0/661819691' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T13:31:45.555 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: mgrmap e17: y(active, starting, since 0.0567234s), standbys: x 2024-05-06T13:31:45.555 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:31:45.555 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:31:45.555 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:31:45.555 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T13:31:45.555 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T13:31:45.555 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:31:45.555 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:31:45.555 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:31:45.555 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:31:45.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:31:45.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:31:45.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:31:45.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:31:45.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T13:31:45.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T13:31:45.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T13:31:45.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: Manager daemon y is now available 2024-05-06T13:31:45.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:31:45.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:31:45.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: Standby manager daemon x restarted 2024-05-06T13:31:45.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: Standby manager daemon x started 2024-05-06T13:31:45.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:31:45.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:31:45.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/661819691' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T13:31:45.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/661819691' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:31:45.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/661819691' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T13:31:45.557 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.170:0/661819691' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:31:45.557 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:45.557 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:31:45.557 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:31:45.557 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:45.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/661819691' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:31:45.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:45.558 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:31:45.558 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:31:45.558 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:45 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:46.313 DEBUG:teuthology.orchestra.run.smithi077:node-exporter.a> sudo journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@node-exporter.a.service 2024-05-06T13:31:46.315 DEBUG:teuthology.orchestra.run.smithi170:node-exporter.b> sudo journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@node-exporter.b.service 2024-05-06T13:31:46.318 INFO:tasks.cephadm:Adding alertmanager.a on smithi077 2024-05-06T13:31:46.318 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph orch apply alertmanager '1;smithi077=a' 2024-05-06T13:31:46.462 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:46 smithi077 ceph-mon[22428]: [06/May/2024:13:31:44] ENGINE Bus STARTING 2024-05-06T13:31:46.462 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:46 smithi077 ceph-mon[22428]: [06/May/2024:13:31:45] ENGINE Serving on https://172.21.15.77:7150 2024-05-06T13:31:46.462 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:46 smithi077 ceph-mon[22428]: [06/May/2024:13:31:45] ENGINE Bus STARTED 2024-05-06T13:31:46.463 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:46 smithi077 ceph-mon[22428]: mgrmap e18: y(active, since 1.07455s), standbys: x 2024-05-06T13:31:46.463 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:46 smithi077 ceph-mon[22428]: from='client.34331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi077=a;smithi170=b", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:31:46.463 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:46 smithi077 ceph-mon[22428]: Saving service node-exporter spec with placement smithi077=a;smithi170=b;count:2 2024-05-06T13:31:46.463 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:46 smithi077 ceph-mon[22428]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:46.463 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:46 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:46.463 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:46 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:46.463 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:46 smithi077 ceph-mon[27166]: [06/May/2024:13:31:44] ENGINE Bus STARTING 2024-05-06T13:31:46.463 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:46 smithi077 ceph-mon[27166]: [06/May/2024:13:31:45] ENGINE Serving on https://172.21.15.77:7150 2024-05-06T13:31:46.464 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:46 smithi077 ceph-mon[27166]: [06/May/2024:13:31:45] ENGINE Bus STARTED 2024-05-06T13:31:46.464 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:46 smithi077 ceph-mon[27166]: mgrmap e18: y(active, since 1.07455s), standbys: x 2024-05-06T13:31:46.464 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:46 smithi077 ceph-mon[27166]: from='client.34331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi077=a;smithi170=b", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:31:46.464 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:46 smithi077 ceph-mon[27166]: Saving service node-exporter spec with placement smithi077=a;smithi170=b;count:2 2024-05-06T13:31:46.464 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:46 smithi077 ceph-mon[27166]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:46.464 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:46 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:46.464 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:46 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:46.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:46 smithi170 ceph-mon[23272]: [06/May/2024:13:31:44] ENGINE Bus STARTING 2024-05-06T13:31:46.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:46 smithi170 ceph-mon[23272]: [06/May/2024:13:31:45] ENGINE Serving on https://172.21.15.77:7150 2024-05-06T13:31:46.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:46 smithi170 ceph-mon[23272]: [06/May/2024:13:31:45] ENGINE Bus STARTED 2024-05-06T13:31:46.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:46 smithi170 ceph-mon[23272]: mgrmap e18: y(active, since 1.07455s), standbys: x 2024-05-06T13:31:46.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:46 smithi170 ceph-mon[23272]: from='client.34331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi077=a;smithi170=b", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:31:46.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:46 smithi170 ceph-mon[23272]: Saving service node-exporter spec with placement smithi077=a;smithi170=b;count:2 2024-05-06T13:31:46.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:46 smithi170 ceph-mon[23272]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:46.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:46 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:46.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:46 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:47.489 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:47 smithi170 ceph-mon[23272]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:47.490 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:47 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:47.490 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:47 smithi170 ceph-mon[23272]: Updating smithi077:/etc/ceph/ceph.conf 2024-05-06T13:31:47.490 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:47 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:47.490 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:47 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:47.490 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:47 smithi170 ceph-mon[23272]: mgrmap e19: y(active, since 2s), standbys: x 2024-05-06T13:31:47.490 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:47 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:47.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:47 smithi077 ceph-mon[22428]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:47.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:47 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:47.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:47 smithi077 ceph-mon[22428]: Updating smithi077:/etc/ceph/ceph.conf 2024-05-06T13:31:47.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:47 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:47.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:47 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:47.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:47 smithi077 ceph-mon[22428]: mgrmap e19: y(active, since 2s), standbys: x 2024-05-06T13:31:47.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:47 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:47.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:47 smithi077 ceph-mon[27166]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:47.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:47 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:47.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:47 smithi077 ceph-mon[27166]: Updating smithi077:/etc/ceph/ceph.conf 2024-05-06T13:31:47.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:47 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:47.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:47 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:47.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:47 smithi077 ceph-mon[27166]: mgrmap e19: y(active, since 2s), standbys: x 2024-05-06T13:31:47.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:47 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:48.374 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled alertmanager update... 2024-05-06T13:31:48.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:48 smithi077 ceph-mon[22428]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:31:48.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:48 smithi077 ceph-mon[22428]: Updating smithi170:/etc/ceph/ceph.conf 2024-05-06T13:31:48.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:48 smithi077 ceph-mon[22428]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:31:48.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:48 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:48.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:48 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:48.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:48 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:48.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:48 smithi077 ceph-mon[27166]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:31:48.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:48 smithi077 ceph-mon[27166]: Updating smithi170:/etc/ceph/ceph.conf 2024-05-06T13:31:48.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:48 smithi077 ceph-mon[27166]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:31:48.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:48 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:48.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:48 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:48.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:48 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:48.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:48 smithi170 ceph-mon[23272]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:31:48.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:48 smithi170 ceph-mon[23272]: Updating smithi170:/etc/ceph/ceph.conf 2024-05-06T13:31:48.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:48 smithi170 ceph-mon[23272]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:31:48.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:48 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:48.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:48 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:48.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:48 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:48.994 DEBUG:teuthology.orchestra.run.smithi077:alertmanager.a> sudo journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@alertmanager.a.service 2024-05-06T13:31:48.997 INFO:tasks.cephadm:Adding grafana.a on smithi170 2024-05-06T13:31:48.997 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph orch apply grafana '1;smithi170=a' 2024-05-06T13:31:49.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:49 smithi170 ceph-mon[23272]: Deploying daemon prometheus.a on smithi170 2024-05-06T13:31:49.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:49 smithi170 ceph-mon[23272]: from='client.34358 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi077=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:31:49.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:49 smithi170 ceph-mon[23272]: Saving service alertmanager spec with placement smithi077=a;count:1 2024-05-06T13:31:49.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:49 smithi170 ceph-mon[23272]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:49.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:49 smithi077 ceph-mon[22428]: Deploying daemon prometheus.a on smithi170 2024-05-06T13:31:49.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:49 smithi077 ceph-mon[22428]: from='client.34358 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi077=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:31:49.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:49 smithi077 ceph-mon[22428]: Saving service alertmanager spec with placement smithi077=a;count:1 2024-05-06T13:31:49.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:49 smithi077 ceph-mon[22428]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:49.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:49 smithi077 ceph-mon[27166]: Deploying daemon prometheus.a on smithi170 2024-05-06T13:31:49.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:49 smithi077 ceph-mon[27166]: from='client.34358 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi077=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:31:49.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:49 smithi077 ceph-mon[27166]: Saving service alertmanager spec with placement smithi077=a;count:1 2024-05-06T13:31:49.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:49 smithi077 ceph-mon[27166]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:50.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:50 smithi170 ceph-mon[23272]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:50.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:50 smithi077 ceph-mon[22428]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:50.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:50 smithi077 ceph-mon[27166]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:51.892 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled grafana update... 2024-05-06T13:31:52.908 DEBUG:teuthology.orchestra.run.smithi170:grafana.a> sudo journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@grafana.a.service 2024-05-06T13:31:52.910 INFO:tasks.cephadm:Setting up client nodes... 2024-05-06T13:31:52.911 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-06T13:31:53.220 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:52 smithi077 ceph-mon[27166]: from='client.34364 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi170=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:31:53.220 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:52 smithi077 ceph-mon[27166]: Saving service grafana spec with placement smithi170=a;count:1 2024-05-06T13:31:53.220 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:52 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:53.220 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:52 smithi077 ceph-mon[27166]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:53.221 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:52 smithi077 ceph-mon[22428]: from='client.34364 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi170=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:31:53.221 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:52 smithi077 ceph-mon[22428]: Saving service grafana spec with placement smithi170=a;count:1 2024-05-06T13:31:53.221 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:52 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:53.221 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:52 smithi077 ceph-mon[22428]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:53.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:52 smithi170 ceph-mon[23272]: from='client.34364 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi170=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:31:53.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:52 smithi170 ceph-mon[23272]: Saving service grafana spec with placement smithi170=a;count:1 2024-05-06T13:31:53.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:52 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:31:53.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:52 smithi170 ceph-mon[23272]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:54.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:54 smithi077 ceph-mon[22428]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:54.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:54 smithi077 ceph-mon[27166]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:54.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:54 smithi170 ceph-mon[23272]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:54.976 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2024-05-06T13:31:54.976 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDK2zhmuAgDOhAA8GwEVZJI5ayiXZDVRyBmUg== 2024-05-06T13:31:55.702 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:55 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/1599043706' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-06T13:31:55.702 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:55 smithi077 ceph-mon[22428]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-06T13:31:55.702 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:55 smithi077 ceph-mon[22428]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-06T13:31:55.703 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:55 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/1599043706' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-06T13:31:55.703 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:55 smithi077 ceph-mon[27166]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-06T13:31:55.703 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:55 smithi077 ceph-mon[27166]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-06T13:31:55.727 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-05-06T13:31:55.728 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-05-06T13:31:55.728 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-06T13:31:55.767 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-06T13:31:55.789 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:55 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/1599043706' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-06T13:31:55.789 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:55 smithi170 ceph-mon[23272]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-06T13:31:55.789 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:55 smithi170 ceph-mon[23272]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-06T13:31:56.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:56 smithi170 ceph-mon[23272]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:56.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:56 smithi077 ceph-mon[22428]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:56.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:56 smithi077 ceph-mon[27166]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:58.797 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:31:58 smithi170 ceph-mon[23272]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:58.801 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:31:58 smithi077 ceph-mon[22428]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:58.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:31:58 smithi077 ceph-mon[27166]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:31:59.326 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:31:59 smithi170 systemd[1]: Starting Ceph prometheus.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:32:00.239 INFO:teuthology.orchestra.run.smithi170.stdout:[client.1] 2024-05-06T13:32:00.240 INFO:teuthology.orchestra.run.smithi170.stdout: key = AQDQ2zhmaBQODhAASEqjtNCq+WQrtJUF8gl8AQ== 2024-05-06T13:32:00.323 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:32:00 smithi170 podman[38375]: 2024-05-06 13:32:00.072796583 +0000 UTC m=+0.091316934 container create 20bf0e71db759f750d957a896d65e6a398331006eac164c041f1d0dc713ffd59 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:32:00.324 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:32:00 smithi170 podman[38375]: 2024-05-06 13:32:00.000481779 +0000 UTC m=+0.019002141 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-05-06T13:32:00.324 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:32:00 smithi170 podman[38375]: 2024-05-06 13:32:00.292206192 +0000 UTC m=+0.310726543 container init 20bf0e71db759f750d957a896d65e6a398331006eac164c041f1d0dc713ffd59 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:32:00.324 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:32:00 smithi170 podman[38375]: 2024-05-06 13:32:00.295036415 +0000 UTC m=+0.313556778 container start 20bf0e71db759f750d957a896d65e6a398331006eac164c041f1d0dc713ffd59 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:32:00.324 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:32:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:32:00.318Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-05-06T13:32:00.324 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:32:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:32:00.318Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-05-06T13:32:00.324 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:32:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:32:00.318Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-05-06T13:32:00.324 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:32:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:32:00.318Z caller=main.go:518 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi170 (none))" 2024-05-06T13:32:00.324 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:32:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:32:00.318Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-05-06T13:32:00.324 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:32:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:32:00.318Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-05-06T13:32:00.324 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:32:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:32:00.320Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-05-06T13:32:00.324 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:32:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:32:00.320Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-05-06T13:32:00.324 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:32:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:32:00.321Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-05-06T13:32:00.324 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:32:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:32:00.322Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-05-06T13:32:00.325 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:32:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:32:00.322Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.468µs 2024-05-06T13:32:00.325 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:32:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:32:00.322Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-05-06T13:32:00.325 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:32:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:32:00.323Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-05-06T13:32:00.325 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:32:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:32:00.323Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=31.478µs wal_replay_duration=262.493µs total_replay_duration=315.566µs 2024-05-06T13:32:00.325 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:32:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:32:00.323Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-05-06T13:32:00.325 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:32:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:32:00.323Z caller=main.go:947 level=info msg="TSDB started" 2024-05-06T13:32:00.325 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:32:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:32:00.323Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-05-06T13:32:00.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:00 smithi170 ceph-mon[23272]: from='client.? 172.21.15.170:0/3211316547' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-06T13:32:00.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:00 smithi170 ceph-mon[23272]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-06T13:32:00.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:00 smithi170 ceph-mon[23272]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-06T13:32:00.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:00 smithi170 ceph-mon[23272]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:00.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:00 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:00.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:00 smithi170 ceph-mon[23272]: Deploying daemon node-exporter.a on smithi077 2024-05-06T13:32:00.577 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:32:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:32:00.340Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.66315ms db_storage=996ns remote_storage=1.898µs web_handler=300ns query_engine=776ns scrape=1.044196ms scrape_sd=17.309µs notify=860ns notify_sd=1.569µs rules=15.385106ms 2024-05-06T13:32:00.577 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:32:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:32:00.340Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-05-06T13:32:00.577 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:32:00 smithi170 bash[38375]: 20bf0e71db759f750d957a896d65e6a398331006eac164c041f1d0dc713ffd59 2024-05-06T13:32:00.577 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:32:00 smithi170 systemd[1]: Started Ceph prometheus.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:32:00.722 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:00 smithi077 ceph-mon[27166]: from='client.? 172.21.15.170:0/3211316547' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-06T13:32:00.723 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:00 smithi077 ceph-mon[27166]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-06T13:32:00.723 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:00 smithi077 ceph-mon[27166]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-06T13:32:00.723 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:00 smithi077 ceph-mon[27166]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:00.723 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:00 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:00.723 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:00 smithi077 ceph-mon[27166]: Deploying daemon node-exporter.a on smithi077 2024-05-06T13:32:00.723 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:00 smithi077 ceph-mon[22428]: from='client.? 172.21.15.170:0/3211316547' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-06T13:32:00.723 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:00 smithi077 ceph-mon[22428]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-06T13:32:00.723 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:00 smithi077 ceph-mon[22428]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-06T13:32:00.723 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:00 smithi077 ceph-mon[22428]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:00.724 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:00 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:00.724 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:00 smithi077 ceph-mon[22428]: Deploying daemon node-exporter.a on smithi077 2024-05-06T13:32:01.034 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-05-06T13:32:01.034 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-05-06T13:32:01.034 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-05-06T13:32:01.110 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-06T13:32:01.110 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-05-06T13:32:01.110 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph mgr dump --format=json 2024-05-06T13:32:01.337 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T13:32:01.459 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:01 smithi077 bash[41945]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-05-06T13:32:02.052 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:01 smithi077 bash[41945]: Getting image source signatures 2024-05-06T13:32:02.052 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:01 smithi077 bash[41945]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-05-06T13:32:02.052 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:01 smithi077 bash[41945]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-05-06T13:32:02.052 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:01 smithi077 bash[41945]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-05-06T13:32:02.801 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:02 smithi077 ceph-mon[22428]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:02.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:02 smithi077 ceph-mon[27166]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:02.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:02 smithi170 ceph-mon[23272]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:03.644 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:03 smithi077 bash[41945]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-05-06T13:32:03.645 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:03 smithi077 bash[41945]: Writing manifest to image destination 2024-05-06T13:32:04.060 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:32:04.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:04 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/1496449608' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-06T13:32:04.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:04 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/1496449608' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-06T13:32:04.303 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:04 smithi077 podman[41945]: 2024-05-06 13:32:04.009353908 +0000 UTC m=+2.825982397 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-05-06T13:32:04.303 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:04 smithi077 podman[41945]: 2024-05-06 13:32:04.083014214 +0000 UTC m=+2.899642696 container create eb6f92239232c4dd81c6a40f5707002d95388ca9e2f9c755195fa1af8c575dfc (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T13:32:04.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:04 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/1496449608' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-06T13:32:04.896 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":19,"active_gid":24352,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2284839445},{"type":"v1","addr":"172.21.15.77:6801","nonce":2284839445}]},"active_addr":"172.21.15.77:6801/2284839445","active_change":"2024-05-06T13:31:44.360812+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":34319,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.77:8443/","prometheus":"http://172.21.15.77:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":60,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":2214497969}]},{"addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":1241103129}]},{"addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":2331082954}]},{"addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":930296124}]}]}} 2024-05-06T13:32:04.902 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-05-06T13:32:04.902 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-06T13:32:04.902 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-06T13:32:05.123 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-mon[22428]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:05.123 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:05.123 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:05 smithi077 ceph-mon[27166]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:05.123 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:05 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:05.124 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 podman[41945]: 2024-05-06 13:32:05.023072371 +0000 UTC m=+3.839700855 container init eb6f92239232c4dd81c6a40f5707002d95388ca9e2f9c755195fa1af8c575dfc (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T13:32:05.124 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 podman[41945]: 2024-05-06 13:32:05.026862707 +0000 UTC m=+3.843491197 container start eb6f92239232c4dd81c6a40f5707002d95388ca9e2f9c755195fa1af8c575dfc (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T13:32:05.124 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.030Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-05-06T13:32:05.124 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.030Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-05-06T13:32:05.124 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.030Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-05-06T13:32:05.124 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.030Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-05-06T13:32:05.124 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-05-06T13:32:05.124 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=arp 2024-05-06T13:32:05.124 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=bcache 2024-05-06T13:32:05.124 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=bonding 2024-05-06T13:32:05.124 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=btrfs 2024-05-06T13:32:05.124 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=conntrack 2024-05-06T13:32:05.124 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=cpu 2024-05-06T13:32:05.124 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-05-06T13:32:05.124 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=diskstats 2024-05-06T13:32:05.125 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=dmi 2024-05-06T13:32:05.125 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=edac 2024-05-06T13:32:05.125 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=entropy 2024-05-06T13:32:05.125 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-05-06T13:32:05.125 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=filefd 2024-05-06T13:32:05.125 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=filesystem 2024-05-06T13:32:05.125 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=hwmon 2024-05-06T13:32:05.125 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=infiniband 2024-05-06T13:32:05.125 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=ipvs 2024-05-06T13:32:05.125 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=loadavg 2024-05-06T13:32:05.125 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=mdadm 2024-05-06T13:32:05.125 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=meminfo 2024-05-06T13:32:05.125 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=netclass 2024-05-06T13:32:05.125 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=netdev 2024-05-06T13:32:05.125 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=netstat 2024-05-06T13:32:05.126 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=nfs 2024-05-06T13:32:05.126 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=nfsd 2024-05-06T13:32:05.126 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=nvme 2024-05-06T13:32:05.126 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=os 2024-05-06T13:32:05.126 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-05-06T13:32:05.126 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=pressure 2024-05-06T13:32:05.126 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=rapl 2024-05-06T13:32:05.126 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=schedstat 2024-05-06T13:32:05.126 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=sockstat 2024-05-06T13:32:05.126 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=softnet 2024-05-06T13:32:05.127 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=stat 2024-05-06T13:32:05.127 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=tapestats 2024-05-06T13:32:05.127 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=textfile 2024-05-06T13:32:05.127 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-05-06T13:32:05.127 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=time 2024-05-06T13:32:05.127 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-05-06T13:32:05.127 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=uname 2024-05-06T13:32:05.127 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=vmstat 2024-05-06T13:32:05.127 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=xfs 2024-05-06T13:32:05.127 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:115 level=info collector=zfs 2024-05-06T13:32:05.127 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.031Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-05-06T13:32:05.127 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[42114]: ts=2024-05-06T13:32:05.032Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-05-06T13:32:05.127 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 bash[41945]: eb6f92239232c4dd81c6a40f5707002d95388ca9e2f9c755195fa1af8c575dfc 2024-05-06T13:32:05.176 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T13:32:05.423 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:32:05 smithi077 systemd[1]: Started Ceph node-exporter.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:32:05.476 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:05 smithi170 ceph-mon[23272]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:05.476 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:05 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:06.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:06 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:06.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:06 smithi077 ceph-mon[27166]: Deploying daemon node-exporter.b on smithi170 2024-05-06T13:32:06.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:06 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:06.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:06 smithi077 ceph-mon[22428]: Deploying daemon node-exporter.b on smithi170 2024-05-06T13:32:06.462 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:06 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:06.462 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:06 smithi170 ceph-mon[23272]: Deploying daemon node-exporter.b on smithi170 2024-05-06T13:32:06.826 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:06 smithi170 bash[38655]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-05-06T13:32:07.014 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:32:07.014 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":60,"fsid":"20e59cb4-0bac-11ef-bc97-c7b262605968","created":"2024-05-06T13:26:37.391183+0000","modified":"2024-05-06T13:31:44.360472+0000","last_up_change":"2024-05-06T13:31:26.464116+0000","last_in_change":"2024-05-06T13:31:08.958161+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-06T13:29:32.660551+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"8baf11f2-252b-4937-8684-28c621b69971","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":834793539},{"type":"v1","addr":"172.21.15.77:6803","nonce":834793539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":834793539},{"type":"v1","addr":"172.21.15.77:6805","nonce":834793539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":834793539},{"type":"v1","addr":"172.21.15.77:6809","nonce":834793539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":834793539},{"type":"v1","addr":"172.21.15.77:6807","nonce":834793539}]},"public_addr":"172.21.15.77:6803/834793539","cluster_addr":"172.21.15.77:6805/834793539","heartbeat_back_addr":"172.21.15.77:6809/834793539","heartbeat_front_addr":"172.21.15.77:6807/834793539","state":["exists","up"]},{"osd":1,"uuid":"69b4ced2-2786-41d0-be2d-2acfcb52ece1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2133267326},{"type":"v1","addr":"172.21.15.77:6811","nonce":2133267326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2133267326},{"type":"v1","addr":"172.21.15.77:6813","nonce":2133267326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":2133267326},{"type":"v1","addr":"172.21.15.77:6817","nonce":2133267326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2133267326},{"type":"v1","addr":"172.21.15.77:6815","nonce":2133267326}]},"public_addr":"172.21.15.77:6811/2133267326","cluster_addr":"172.21.15.77:6813/2133267326","heartbeat_back_addr":"172.21.15.77:6817/2133267326","heartbeat_front_addr":"172.21.15.77:6815/2133267326","state":["exists","up"]},{"osd":2,"uuid":"dcab9e69-bf02-4b9f-9ff1-5dbb5d71a557","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":2522642178},{"type":"v1","addr":"172.21.15.77:6819","nonce":2522642178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":2522642178},{"type":"v1","addr":"172.21.15.77:6821","nonce":2522642178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2522642178},{"type":"v1","addr":"172.21.15.77:6825","nonce":2522642178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":2522642178},{"type":"v1","addr":"172.21.15.77:6823","nonce":2522642178}]},"public_addr":"172.21.15.77:6819/2522642178","cluster_addr":"172.21.15.77:6821/2522642178","heartbeat_back_addr":"172.21.15.77:6825/2522642178","heartbeat_front_addr":"172.21.15.77:6823/2522642178","state":["exists","up"]},{"osd":3,"uuid":"4b177913-e714-4f93-99b4-e2da20777d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3252753833},{"type":"v1","addr":"172.21.15.77:6827","nonce":3252753833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3252753833},{"type":"v1","addr":"172.21.15.77:6829","nonce":3252753833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3252753833},{"type":"v1","addr":"172.21.15.77:6833","nonce":3252753833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3252753833},{"type":"v1","addr":"172.21.15.77:6831","nonce":3252753833}]},"public_addr":"172.21.15.77:6827/3252753833","cluster_addr":"172.21.15.77:6829/3252753833","heartbeat_back_addr":"172.21.15.77:6833/3252753833","heartbeat_front_addr":"172.21.15.77:6831/3252753833","state":["exists","up"]},{"osd":4,"uuid":"ce6d16fa-5c8c-4a94-852b-f8193d47d25f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":4006535878},{"type":"v1","addr":"172.21.15.170:6801","nonce":4006535878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":4006535878},{"type":"v1","addr":"172.21.15.170:6803","nonce":4006535878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":4006535878},{"type":"v1","addr":"172.21.15.170:6807","nonce":4006535878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":4006535878},{"type":"v1","addr":"172.21.15.170:6805","nonce":4006535878}]},"public_addr":"172.21.15.170:6801/4006535878","cluster_addr":"172.21.15.170:6803/4006535878","heartbeat_back_addr":"172.21.15.170:6807/4006535878","heartbeat_front_addr":"172.21.15.170:6805/4006535878","state":["exists","up"]},{"osd":5,"uuid":"4f0bd2da-d332-4d55-ad70-86bde46189ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":186430802},{"type":"v1","addr":"172.21.15.170:6809","nonce":186430802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":186430802},{"type":"v1","addr":"172.21.15.170:6811","nonce":186430802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":186430802},{"type":"v1","addr":"172.21.15.170:6815","nonce":186430802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":186430802},{"type":"v1","addr":"172.21.15.170:6813","nonce":186430802}]},"public_addr":"172.21.15.170:6809/186430802","cluster_addr":"172.21.15.170:6811/186430802","heartbeat_back_addr":"172.21.15.170:6815/186430802","heartbeat_front_addr":"172.21.15.170:6813/186430802","state":["exists","up"]},{"osd":6,"uuid":"2f6caf5a-03a5-4151-a129-90f77779e3ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1050047795},{"type":"v1","addr":"172.21.15.170:6817","nonce":1050047795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1050047795},{"type":"v1","addr":"172.21.15.170:6819","nonce":1050047795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1050047795},{"type":"v1","addr":"172.21.15.170:6823","nonce":1050047795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1050047795},{"type":"v1","addr":"172.21.15.170:6821","nonce":1050047795}]},"public_addr":"172.21.15.170:6817/1050047795","cluster_addr":"172.21.15.170:6819/1050047795","heartbeat_back_addr":"172.21.15.170:6823/1050047795","heartbeat_front_addr":"172.21.15.170:6821/1050047795","state":["exists","up"]},{"osd":7,"uuid":"4f0ec75a-6198-4200-b4ea-4f6421b7854b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":604573004},{"type":"v1","addr":"172.21.15.170:6825","nonce":604573004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":604573004},{"type":"v1","addr":"172.21.15.170:6827","nonce":604573004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":604573004},{"type":"v1","addr":"172.21.15.170:6831","nonce":604573004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":604573004},{"type":"v1","addr":"172.21.15.170:6829","nonce":604573004}]},"public_addr":"172.21.15.170:6825/604573004","cluster_addr":"172.21.15.170:6827/604573004","heartbeat_back_addr":"172.21.15.170:6831/604573004","heartbeat_front_addr":"172.21.15.170:6829/604573004","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T13:28:42.270920+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T13:29:05.303372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T13:29:29.095238+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T13:29:53.475953+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T13:30:15.076693+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T13:30:38.107348+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T13:31:01.288168+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T13:31:24.484479+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1194062530":"2024-05-07T13:31:44.360425+0000","172.21.15.77:0/3607570357":"2024-05-07T13:31:44.360425+0000","172.21.15.77:6800/3988065694":"2024-05-07T13:31:44.360425+0000","172.21.15.77:0/371454062":"2024-05-07T13:31:44.360425+0000","172.21.15.77:6801/3988065694":"2024-05-07T13:31:44.360425+0000","172.21.15.77:0/4134859885":"2024-05-07T13:31:44.360425+0000","172.21.15.77:6801/1922446457":"2024-05-07T13:27:24.299634+0000","172.21.15.77:0/554686488":"2024-05-07T13:27:24.299634+0000","172.21.15.77:0/3193136642":"2024-05-07T13:27:02.199372+0000","172.21.15.77:0/2039164702":"2024-05-07T13:27:02.199372+0000","172.21.15.77:0/1112825224":"2024-05-07T13:27:24.299634+0000","172.21.15.77:6800/2007005358":"2024-05-07T13:27:02.199372+0000","172.21.15.77:6800/1922446457":"2024-05-07T13:27:24.299634+0000","172.21.15.77:0/3698539339":"2024-05-07T13:27:24.299634+0000","172.21.15.77:6801/2007005358":"2024-05-07T13:27:02.199372+0000","172.21.15.77:0/461831909":"2024-05-07T13:27:02.199372+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-06T13:32:07.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:07 smithi077 ceph-mon[27166]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:07.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:07 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/1705740447' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T13:32:07.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:07 smithi077 ceph-mon[22428]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:07.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:07 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/1705740447' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T13:32:07.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:07 smithi170 ceph-mon[23272]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:07.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:07 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/1705740447' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T13:32:07.719 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-06T13:32:07.720 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-06T13:32:07.928 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T13:32:08.076 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:07 smithi170 bash[38655]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-05-06T13:32:08.076 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:07 smithi170 bash[38655]: Writing manifest to image destination 2024-05-06T13:32:08.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-mon[23272]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:08.575 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 podman[38655]: 2024-05-06 13:32:08.264816635 +0000 UTC m=+2.294670653 container create 001e61e5f9228963396904bf202c8180726e112d99ff8ab1167901230630367d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T13:32:08.575 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 podman[38655]: 2024-05-06 13:32:08.199445963 +0000 UTC m=+2.229299994 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-05-06T13:32:08.575 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 podman[38655]: 2024-05-06 13:32:08.466656813 +0000 UTC m=+2.496510831 container init 001e61e5f9228963396904bf202c8180726e112d99ff8ab1167901230630367d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T13:32:08.575 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 podman[38655]: 2024-05-06 13:32:08.469387008 +0000 UTC m=+2.499241031 container start 001e61e5f9228963396904bf202c8180726e112d99ff8ab1167901230630367d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T13:32:08.575 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.473Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-05-06T13:32:08.575 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.473Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-05-06T13:32:08.575 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.473Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-05-06T13:32:08.575 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.473Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-05-06T13:32:08.575 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-05-06T13:32:08.576 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=arp 2024-05-06T13:32:08.576 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=bcache 2024-05-06T13:32:08.576 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=bonding 2024-05-06T13:32:08.576 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=btrfs 2024-05-06T13:32:08.576 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=conntrack 2024-05-06T13:32:08.576 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=cpu 2024-05-06T13:32:08.576 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-05-06T13:32:08.577 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=diskstats 2024-05-06T13:32:08.577 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=dmi 2024-05-06T13:32:08.577 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=edac 2024-05-06T13:32:08.577 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=entropy 2024-05-06T13:32:08.577 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-05-06T13:32:08.577 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=filefd 2024-05-06T13:32:08.577 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=filesystem 2024-05-06T13:32:08.577 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=hwmon 2024-05-06T13:32:08.577 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=infiniband 2024-05-06T13:32:08.577 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=ipvs 2024-05-06T13:32:08.577 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=loadavg 2024-05-06T13:32:08.578 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=mdadm 2024-05-06T13:32:08.578 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=meminfo 2024-05-06T13:32:08.578 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=netclass 2024-05-06T13:32:08.578 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=netdev 2024-05-06T13:32:08.578 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=netstat 2024-05-06T13:32:08.578 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=nfs 2024-05-06T13:32:08.578 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=nfsd 2024-05-06T13:32:08.578 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=nvme 2024-05-06T13:32:08.578 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=os 2024-05-06T13:32:08.578 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-05-06T13:32:08.578 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=pressure 2024-05-06T13:32:08.578 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=rapl 2024-05-06T13:32:08.579 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=schedstat 2024-05-06T13:32:08.579 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=sockstat 2024-05-06T13:32:08.579 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=softnet 2024-05-06T13:32:08.579 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=stat 2024-05-06T13:32:08.579 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=tapestats 2024-05-06T13:32:08.579 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=textfile 2024-05-06T13:32:08.579 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-05-06T13:32:08.579 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=time 2024-05-06T13:32:08.579 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-05-06T13:32:08.579 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=uname 2024-05-06T13:32:08.579 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=vmstat 2024-05-06T13:32:08.579 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=xfs 2024-05-06T13:32:08.579 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:115 level=info collector=zfs 2024-05-06T13:32:08.580 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.474Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-05-06T13:32:08.580 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[38715]: ts=2024-05-06T13:32:08.475Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-05-06T13:32:08.580 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 bash[38655]: 001e61e5f9228963396904bf202c8180726e112d99ff8ab1167901230630367d 2024-05-06T13:32:08.665 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:08 smithi077 ceph-mon[22428]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:08.665 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:08 smithi077 ceph-mon[27166]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:08.826 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:32:08 smithi170 systemd[1]: Started Ceph node-exporter.b for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:32:09.805 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:32:09.805 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":60,"fsid":"20e59cb4-0bac-11ef-bc97-c7b262605968","created":"2024-05-06T13:26:37.391183+0000","modified":"2024-05-06T13:31:44.360472+0000","last_up_change":"2024-05-06T13:31:26.464116+0000","last_in_change":"2024-05-06T13:31:08.958161+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-06T13:29:32.660551+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"8baf11f2-252b-4937-8684-28c621b69971","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":834793539},{"type":"v1","addr":"172.21.15.77:6803","nonce":834793539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":834793539},{"type":"v1","addr":"172.21.15.77:6805","nonce":834793539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":834793539},{"type":"v1","addr":"172.21.15.77:6809","nonce":834793539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":834793539},{"type":"v1","addr":"172.21.15.77:6807","nonce":834793539}]},"public_addr":"172.21.15.77:6803/834793539","cluster_addr":"172.21.15.77:6805/834793539","heartbeat_back_addr":"172.21.15.77:6809/834793539","heartbeat_front_addr":"172.21.15.77:6807/834793539","state":["exists","up"]},{"osd":1,"uuid":"69b4ced2-2786-41d0-be2d-2acfcb52ece1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":2133267326},{"type":"v1","addr":"172.21.15.77:6811","nonce":2133267326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":2133267326},{"type":"v1","addr":"172.21.15.77:6813","nonce":2133267326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":2133267326},{"type":"v1","addr":"172.21.15.77:6817","nonce":2133267326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":2133267326},{"type":"v1","addr":"172.21.15.77:6815","nonce":2133267326}]},"public_addr":"172.21.15.77:6811/2133267326","cluster_addr":"172.21.15.77:6813/2133267326","heartbeat_back_addr":"172.21.15.77:6817/2133267326","heartbeat_front_addr":"172.21.15.77:6815/2133267326","state":["exists","up"]},{"osd":2,"uuid":"dcab9e69-bf02-4b9f-9ff1-5dbb5d71a557","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":2522642178},{"type":"v1","addr":"172.21.15.77:6819","nonce":2522642178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":2522642178},{"type":"v1","addr":"172.21.15.77:6821","nonce":2522642178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":2522642178},{"type":"v1","addr":"172.21.15.77:6825","nonce":2522642178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":2522642178},{"type":"v1","addr":"172.21.15.77:6823","nonce":2522642178}]},"public_addr":"172.21.15.77:6819/2522642178","cluster_addr":"172.21.15.77:6821/2522642178","heartbeat_back_addr":"172.21.15.77:6825/2522642178","heartbeat_front_addr":"172.21.15.77:6823/2522642178","state":["exists","up"]},{"osd":3,"uuid":"4b177913-e714-4f93-99b4-e2da20777d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3252753833},{"type":"v1","addr":"172.21.15.77:6827","nonce":3252753833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3252753833},{"type":"v1","addr":"172.21.15.77:6829","nonce":3252753833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6832","nonce":3252753833},{"type":"v1","addr":"172.21.15.77:6833","nonce":3252753833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6830","nonce":3252753833},{"type":"v1","addr":"172.21.15.77:6831","nonce":3252753833}]},"public_addr":"172.21.15.77:6827/3252753833","cluster_addr":"172.21.15.77:6829/3252753833","heartbeat_back_addr":"172.21.15.77:6833/3252753833","heartbeat_front_addr":"172.21.15.77:6831/3252753833","state":["exists","up"]},{"osd":4,"uuid":"ce6d16fa-5c8c-4a94-852b-f8193d47d25f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":4006535878},{"type":"v1","addr":"172.21.15.170:6801","nonce":4006535878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":4006535878},{"type":"v1","addr":"172.21.15.170:6803","nonce":4006535878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":4006535878},{"type":"v1","addr":"172.21.15.170:6807","nonce":4006535878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":4006535878},{"type":"v1","addr":"172.21.15.170:6805","nonce":4006535878}]},"public_addr":"172.21.15.170:6801/4006535878","cluster_addr":"172.21.15.170:6803/4006535878","heartbeat_back_addr":"172.21.15.170:6807/4006535878","heartbeat_front_addr":"172.21.15.170:6805/4006535878","state":["exists","up"]},{"osd":5,"uuid":"4f0bd2da-d332-4d55-ad70-86bde46189ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":186430802},{"type":"v1","addr":"172.21.15.170:6809","nonce":186430802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":186430802},{"type":"v1","addr":"172.21.15.170:6811","nonce":186430802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":186430802},{"type":"v1","addr":"172.21.15.170:6815","nonce":186430802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":186430802},{"type":"v1","addr":"172.21.15.170:6813","nonce":186430802}]},"public_addr":"172.21.15.170:6809/186430802","cluster_addr":"172.21.15.170:6811/186430802","heartbeat_back_addr":"172.21.15.170:6815/186430802","heartbeat_front_addr":"172.21.15.170:6813/186430802","state":["exists","up"]},{"osd":6,"uuid":"2f6caf5a-03a5-4151-a129-90f77779e3ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1050047795},{"type":"v1","addr":"172.21.15.170:6817","nonce":1050047795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1050047795},{"type":"v1","addr":"172.21.15.170:6819","nonce":1050047795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1050047795},{"type":"v1","addr":"172.21.15.170:6823","nonce":1050047795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1050047795},{"type":"v1","addr":"172.21.15.170:6821","nonce":1050047795}]},"public_addr":"172.21.15.170:6817/1050047795","cluster_addr":"172.21.15.170:6819/1050047795","heartbeat_back_addr":"172.21.15.170:6823/1050047795","heartbeat_front_addr":"172.21.15.170:6821/1050047795","state":["exists","up"]},{"osd":7,"uuid":"4f0ec75a-6198-4200-b4ea-4f6421b7854b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":604573004},{"type":"v1","addr":"172.21.15.170:6825","nonce":604573004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":604573004},{"type":"v1","addr":"172.21.15.170:6827","nonce":604573004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":604573004},{"type":"v1","addr":"172.21.15.170:6831","nonce":604573004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":604573004},{"type":"v1","addr":"172.21.15.170:6829","nonce":604573004}]},"public_addr":"172.21.15.170:6825/604573004","cluster_addr":"172.21.15.170:6827/604573004","heartbeat_back_addr":"172.21.15.170:6831/604573004","heartbeat_front_addr":"172.21.15.170:6829/604573004","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T13:28:42.270920+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T13:29:05.303372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T13:29:29.095238+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T13:29:53.475953+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T13:30:15.076693+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T13:30:38.107348+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T13:31:01.288168+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-05-06T13:31:24.484479+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/1194062530":"2024-05-07T13:31:44.360425+0000","172.21.15.77:0/3607570357":"2024-05-07T13:31:44.360425+0000","172.21.15.77:6800/3988065694":"2024-05-07T13:31:44.360425+0000","172.21.15.77:0/371454062":"2024-05-07T13:31:44.360425+0000","172.21.15.77:6801/3988065694":"2024-05-07T13:31:44.360425+0000","172.21.15.77:0/4134859885":"2024-05-07T13:31:44.360425+0000","172.21.15.77:6801/1922446457":"2024-05-07T13:27:24.299634+0000","172.21.15.77:0/554686488":"2024-05-07T13:27:24.299634+0000","172.21.15.77:0/3193136642":"2024-05-07T13:27:02.199372+0000","172.21.15.77:0/2039164702":"2024-05-07T13:27:02.199372+0000","172.21.15.77:0/1112825224":"2024-05-07T13:27:24.299634+0000","172.21.15.77:6800/2007005358":"2024-05-07T13:27:02.199372+0000","172.21.15.77:6800/1922446457":"2024-05-07T13:27:24.299634+0000","172.21.15.77:0/3698539339":"2024-05-07T13:27:24.299634+0000","172.21.15.77:6801/2007005358":"2024-05-07T13:27:02.199372+0000","172.21.15.77:0/461831909":"2024-05-07T13:27:02.199372+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-06T13:32:10.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:09 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:10.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:09 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:32:10.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:09 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:32:10.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:09 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:32:10.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:09 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:10.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:09 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:10.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:09 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:32:10.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:09 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:32:10.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:09 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:32:10.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:09 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:10.054 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:09 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:10.054 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:09 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:32:10.054 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:09 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:32:10.054 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:09 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:32:10.054 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:09 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:10.543 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-06T13:32:10.544 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-06T13:32:10.544 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-06T13:32:10.544 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-06T13:32:10.545 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-06T13:32:10.545 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-06T13:32:10.546 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-05-06T13:32:10.546 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-05-06T13:32:10.901 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:10 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/3646730585' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T13:32:10.901 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:10 smithi170 ceph-mon[23272]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:11.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:10 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3646730585' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T13:32:11.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:10 smithi077 ceph-mon[22428]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:11.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:10 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3646730585' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-06T13:32:11.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:10 smithi077 ceph-mon[27166]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:11.123 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T13:32:11.129 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T13:32:11.129 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T13:32:11.129 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T13:32:11.130 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T13:32:11.130 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T13:32:12.425 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:12 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:12.425 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:12 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:12.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:12 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:13.473 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:13 smithi077 ceph-mon[22428]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:13.473 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:13 smithi077 ceph-mon[27166]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:13.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:13 smithi170 ceph-mon[23272]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:14.325 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:32:14 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:32:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:32:14.662 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:14 smithi077 ceph-mon[22428]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:14.662 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:14 smithi077 ceph-mon[27166]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:14.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:14 smithi170 ceph-mon[23272]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:14.916 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:32:14 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:32:14] "GET /metrics HTTP/1.1" 200 191484 "" "Prometheus/2.33.4" 2024-05-06T13:32:16.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:16 smithi077 ceph-mon[22428]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:16.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:16 smithi077 ceph-mon[27166]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:16.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:16 smithi170 ceph-mon[23272]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:18.789 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:18 smithi077 ceph-mon[22428]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:18.789 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:18 smithi077 ceph-mon[27166]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:18.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:18 smithi170 ceph-mon[23272]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:19.239 INFO:teuthology.orchestra.run.smithi077.stdout:167503724564 2024-05-06T13:32:19.239 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-06T13:32:20.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:20 smithi077 ceph-mon[22428]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:20.760 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:20 smithi077 ceph-mon[27166]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:20.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:20 smithi170 ceph-mon[23272]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:21.161 INFO:teuthology.orchestra.run.smithi077.stdout:55834574889 2024-05-06T13:32:21.162 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-06T13:32:21.937 INFO:teuthology.orchestra.run.smithi077.stdout:111669149726 2024-05-06T13:32:21.937 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-06T13:32:22.511 INFO:teuthology.orchestra.run.smithi077.stdout:236223201292 2024-05-06T13:32:22.511 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-05-06T13:32:22.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:22 smithi077 ceph-mon[22428]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:22.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:22 smithi077 ceph-mon[27166]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:22.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:22 smithi170 ceph-mon[23272]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:23.163 INFO:teuthology.orchestra.run.smithi077.stdout:77309411365 2024-05-06T13:32:23.163 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-06T13:32:23.506 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T13:32:23.507 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T13:32:23.507 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T13:32:23.508 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T13:32:23.509 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T13:32:24.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:32:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:32:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:32:24.802 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:32:24 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:32:24] "GET /metrics HTTP/1.1" 200 191484 "" "Prometheus/2.33.4" 2024-05-06T13:32:25.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:25 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:25.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:25 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:25.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:25 smithi077 ceph-mon[22428]: Deploying daemon alertmanager.a on smithi077 2024-05-06T13:32:25.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:25 smithi077 ceph-mon[22428]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:25.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:25 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:25.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:25 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:25.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:25 smithi077 ceph-mon[27166]: Deploying daemon alertmanager.a on smithi077 2024-05-06T13:32:25.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:25 smithi077 ceph-mon[27166]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:25.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:25 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:25.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:25 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:25.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:25 smithi170 ceph-mon[23272]: Deploying daemon alertmanager.a on smithi077 2024-05-06T13:32:25.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:25 smithi170 ceph-mon[23272]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:26.790 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:26 smithi077 ceph-mon[22428]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:26.790 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:26 smithi077 ceph-mon[27166]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:26.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:26 smithi170 ceph-mon[23272]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:28.700 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:28 smithi077 ceph-mon[22428]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:28.700 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:28 smithi077 ceph-mon[27166]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:28.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:28 smithi170 ceph-mon[23272]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:30.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:30 smithi077 ceph-mon[22428]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:30.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:30 smithi077 ceph-mon[27166]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:30.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:30 smithi170 ceph-mon[23272]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:31.564 INFO:teuthology.orchestra.run.smithi077.stdout:201863462931 2024-05-06T13:32:31.565 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-05-06T13:32:31.783 INFO:teuthology.orchestra.run.smithi077.stdout:111669149728 2024-05-06T13:32:32.024 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:31 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/250900480' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-06T13:32:32.025 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:31 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/250900480' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-06T13:32:32.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:31 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/250900480' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-06T13:32:32.824 INFO:teuthology.orchestra.run.smithi077.stdout:133143986203 2024-05-06T13:32:32.825 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-06T13:32:33.004 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:32 smithi077 ceph-mon[22428]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:33.005 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:32 smithi077 ceph-mon[27166]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:33.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:32 smithi170 ceph-mon[23272]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:34.263 INFO:teuthology.orchestra.run.smithi077.stdout:236223201295 2024-05-06T13:32:34.315 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:32:34 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:32:34] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:32:34.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:34 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/778002290' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-06T13:32:34.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:34 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/778002290' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-06T13:32:34.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:34 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/778002290' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-06T13:32:34.853 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:32:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:32:34] "GET /metrics HTTP/1.1" 200 191484 "" "Prometheus/2.33.4" 2024-05-06T13:32:35.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:35 smithi077 ceph-mon[22428]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:35.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:35 smithi077 ceph-mon[27166]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:35.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:35 smithi170 ceph-mon[23272]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:35.920 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149726 got 111669149728 for osd.3 2024-05-06T13:32:35.920 DEBUG:teuthology.parallel:result is None 2024-05-06T13:32:36.726 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201292 got 236223201295 for osd.7 2024-05-06T13:32:36.726 DEBUG:teuthology.parallel:result is None 2024-05-06T13:32:36.753 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T13:32:36.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:36 smithi077 ceph-mon[22428]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:36.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:36 smithi077 ceph-mon[27166]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:36.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:36 smithi170 ceph-mon[23272]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:38.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:38 smithi077 ceph-mon[22428]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:38.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:38 smithi077 ceph-mon[27166]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:38.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:38 smithi170 ceph-mon[23272]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:38.836 INFO:teuthology.orchestra.run.smithi077.stdout:34359738417 2024-05-06T13:32:38.836 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-06T13:32:39.393 INFO:teuthology.orchestra.run.smithi077.stdout:167503724568 2024-05-06T13:32:39.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:39 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2329747008' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-06T13:32:39.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:39 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2329747008' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-06T13:32:39.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:39 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2329747008' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-06T13:32:39.991 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T13:32:39.991 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T13:32:39.992 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T13:32:39.995 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T13:32:40.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:40 smithi077 ceph-mon[22428]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:40.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:40 smithi077 ceph-mon[27166]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:40.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:40 smithi170 ceph-mon[23272]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:42.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:42 smithi077 ceph-mon[22428]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:42.761 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:42 smithi077 ceph-mon[27166]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:42.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:42 smithi170 ceph-mon[23272]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:44.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:32:44 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:32:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:32:44.769 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:32:44 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:32:44] "GET /metrics HTTP/1.1" 200 191483 "" "Prometheus/2.33.4" 2024-05-06T13:32:44.770 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:44 smithi077 ceph-mon[22428]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:44.770 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:32:44.770 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:32:44.770 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:44 smithi077 ceph-mon[27166]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:44.770 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:32:44.770 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:32:44.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:44 smithi170 ceph-mon[23272]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:44.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:32:44.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:32:46.282 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724564 got 167503724568 for osd.5 2024-05-06T13:32:46.282 DEBUG:teuthology.parallel:result is None 2024-05-06T13:32:46.434 INFO:teuthology.orchestra.run.smithi077.stdout:55834574894 2024-05-06T13:32:46.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:46 smithi077 ceph-mon[22428]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:46.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:46 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/1038777567' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-06T13:32:46.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:46 smithi077 ceph-mon[27166]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:46.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:46 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/1038777567' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-06T13:32:46.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:46 smithi170 ceph-mon[23272]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:46.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:46 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/1038777567' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-06T13:32:47.775 INFO:teuthology.orchestra.run.smithi077.stdout:77309411370 2024-05-06T13:32:47.974 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:47 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/4217985897' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-06T13:32:47.975 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:47 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/4217985897' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-06T13:32:48.180 INFO:teuthology.orchestra.run.smithi077.stdout:201863462935 2024-05-06T13:32:48.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:47 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/4217985897' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-06T13:32:48.593 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411365 got 77309411370 for osd.2 2024-05-06T13:32:48.593 DEBUG:teuthology.parallel:result is None 2024-05-06T13:32:49.081 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:48 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/9214959' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-06T13:32:49.081 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:48 smithi077 ceph-mon[22428]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:49.082 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:48 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/9214959' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-06T13:32:49.082 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:48 smithi077 ceph-mon[27166]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:49.111 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574889 got 55834574894 for osd.1 2024-05-06T13:32:49.111 DEBUG:teuthology.parallel:result is None 2024-05-06T13:32:49.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:48 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/9214959' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-06T13:32:49.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:48 smithi170 ceph-mon[23272]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:49.875 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:32:49 smithi077 podman[45154]: 2024-05-06 13:32:49.787840403 +0000 UTC m=+1.790593990 container create 3a8b1fe0f41bee0cfe57c89c8d5dab1967c21482d4a9dcce37e659a447aaa5b0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T13:32:50.138 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:32:50 smithi077 podman[45154]: 2024-05-06 13:32:50.098974983 +0000 UTC m=+2.101728570 container init 3a8b1fe0f41bee0cfe57c89c8d5dab1967c21482d4a9dcce37e659a447aaa5b0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T13:32:50.138 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:32:50 smithi077 podman[45154]: 2024-05-06 13:32:50.101459147 +0000 UTC m=+2.104212735 container start 3a8b1fe0f41bee0cfe57c89c8d5dab1967c21482d4a9dcce37e659a447aaa5b0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T13:32:50.138 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:32:50 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[45250]: level=info ts=2024-05-06T13:32:50.111Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-05-06T13:32:50.138 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:32:50 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[45250]: level=info ts=2024-05-06T13:32:50.111Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-05-06T13:32:50.139 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:32:50 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[45250]: level=info ts=2024-05-06T13:32:50.113Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.77 port=9094 2024-05-06T13:32:50.139 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:32:50 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[45250]: level=info ts=2024-05-06T13:32:50.114Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-05-06T13:32:50.139 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:32:50 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[45250]: level=info ts=2024-05-06T13:32:50.138Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-05-06T13:32:50.139 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:32:50 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[45250]: level=info ts=2024-05-06T13:32:50.138Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-05-06T13:32:50.488 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:32:50 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[45250]: level=info ts=2024-05-06T13:32:50.141Z caller=main.go:518 msg=Listening address=:9093 2024-05-06T13:32:50.488 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:32:50 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[45250]: level=info ts=2024-05-06T13:32:50.141Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-05-06T13:32:50.488 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:32:50 smithi077 bash[45154]: 3a8b1fe0f41bee0cfe57c89c8d5dab1967c21482d4a9dcce37e659a447aaa5b0 2024-05-06T13:32:50.488 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:32:50 smithi077 systemd[1]: Started Ceph alertmanager.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:32:50.665 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462931 got 201863462935 for osd.6 2024-05-06T13:32:50.666 DEBUG:teuthology.parallel:result is None 2024-05-06T13:32:50.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:50 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/628839688' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-06T13:32:50.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:50 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:50.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:50 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:50.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:50 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:50.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:50 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-06T13:32:50.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:50 smithi077 ceph-mon[27166]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-06T13:32:50.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:50 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:50.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:50 smithi077 ceph-mon[27166]: Deploying daemon grafana.a on smithi170 2024-05-06T13:32:50.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:50 smithi077 ceph-mon[27166]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:50.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:50 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/628839688' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-06T13:32:50.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:50 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:50.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:50 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:50.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:50 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:50.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:50 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-06T13:32:50.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:50 smithi077 ceph-mon[22428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-06T13:32:50.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:50 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:50.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:50 smithi077 ceph-mon[22428]: Deploying daemon grafana.a on smithi170 2024-05-06T13:32:50.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:50 smithi077 ceph-mon[22428]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:50.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:50 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/628839688' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-06T13:32:50.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:50 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:50.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:50 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:50.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:50 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:50.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:50 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-06T13:32:50.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:50 smithi170 ceph-mon[23272]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-06T13:32:50.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:50 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:50.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:50 smithi170 ceph-mon[23272]: Deploying daemon grafana.a on smithi170 2024-05-06T13:32:50.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:50 smithi170 ceph-mon[23272]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:51.196 INFO:teuthology.orchestra.run.smithi077.stdout:34359738419 2024-05-06T13:32:51.251 INFO:teuthology.orchestra.run.smithi077.stdout:133143986208 2024-05-06T13:32:51.786 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:51 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3080739448' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-06T13:32:51.787 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:51 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3080739448' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-06T13:32:51.811 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738417 got 34359738419 for osd.0 2024-05-06T13:32:51.811 DEBUG:teuthology.parallel:result is None 2024-05-06T13:32:51.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:51 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/3080739448' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-06T13:32:52.487 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:32:52 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[45250]: level=info ts=2024-05-06T13:32:52.114Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000111483s 2024-05-06T13:32:52.516 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986203 got 133143986208 for osd.4 2024-05-06T13:32:52.516 DEBUG:teuthology.parallel:result is None 2024-05-06T13:32:52.516 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-06T13:32:52.516 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph pg dump --format=json 2024-05-06T13:32:52.732 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T13:32:52.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:52 smithi077 ceph-mon[22428]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:52.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:52 smithi077 ceph-mon[27166]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:52.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:52 smithi170 ceph-mon[23272]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:54.325 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:32:54 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:32:54] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:32:54.578 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:32:54.579 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-05-06T13:32:54.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:54 smithi077 ceph-mon[27166]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:54.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:54 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:54.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:54 smithi077 ceph-mon[22428]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:54.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:54 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:54.803 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:32:54 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:32:54] "GET /metrics HTTP/1.1" 200 191483 "" "Prometheus/2.33.4" 2024-05-06T13:32:54.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:54 smithi170 ceph-mon[23272]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:54.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:54 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:32:55.234 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-05-06T13:32:54.433370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48960,"kb_used_data":6144,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912256,"statfs":{"total":767960285184,"available":767910150144,"internally_reserved":0,"allocated":6291456,"data_stored":3510358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002189"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":266,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-06T13:31:44.497666+0000","last_change":"2024-05-06T13:31:30.485998+0000","last_active":"2024-05-06T13:31:44.497666+0000","last_peered":"2024-05-06T13:31:44.497666+0000","last_clean":"2024-05-06T13:31:44.497666+0000","last_became_active":"2024-05-06T13:31:30.485538+0000","last_became_peered":"2024-05-06T13:31:30.485538+0000","last_unstale":"2024-05-06T13:31:44.497666+0000","last_undegraded":"2024-05-06T13:31:44.497666+0000","last_fullsized":"2024-05-06T13:31:44.497666+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T13:29:33.066627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T13:29:33.066627+0000","last_clean_scrub_stamp":"2024-05-06T13:29:33.066627+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T16:18:25.066514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201299,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6096,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739056,"statfs":{"total":95995035648,"available":95988793344,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 13:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.63600000000000001}]},{"osd":1,"last update":"Mon May 6 13:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.68700000000000006}]},{"osd":2,"last update":"Mon May 6 13:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.62}]},{"osd":3,"last update":"Mon May 6 13:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.746}]},{"osd":4,"last update":"Mon May 6 13:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.76100000000000001}]},{"osd":5,"last update":"Mon May 6 13:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.57799999999999996}]},{"osd":6,"last update":"Mon May 6 13:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.53000000000000003}]}]},{"osd":6,"up_from":47,"seq":201863462936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6156,"kb_used_data":1092,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738996,"statfs":{"total":95995035648,"available":95988731904,"internally_reserved":0,"allocated":1118208,"data_stored":769265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 13:32:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.72599999999999998}]},{"osd":1,"last update":"Mon May 6 13:32:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.66500000000000004}]},{"osd":2,"last update":"Mon May 6 13:32:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.62},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":15.939,"5min":15.939,"15min":15.939},"last":0.63600000000000001}]},{"osd":3,"last update":"Mon May 6 13:32:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.69399999999999995}]},{"osd":4,"last update":"Mon May 6 13:32:05 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.436},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.48899999999999999}]},{"osd":5,"last update":"Mon May 6 13:32:05 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.54500000000000004}]},{"osd":7,"last update":"Mon May 6 13:32:30 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.56999999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6340,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738812,"statfs":{"total":95995035648,"available":95988543488,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 13:32:16 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.33800000000000002,"15min":0.32100000000000001},"min":{"1min":0.251,"5min":0.216,"15min":0.216},"max":{"1min":0.55800000000000005,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.377,"5min":0.35599999999999998,"15min":0.32900000000000001},"min":{"1min":0.23599999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.56999999999999995,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.29299999999999998}]},{"osd":2,"last update":"Mon May 6 13:32:40 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.35599999999999998,"15min":0.35199999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.378,"5min":0.40100000000000002,"15min":0.39500000000000002},"min":{"1min":0.26100000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.36499999999999999}]},{"osd":3,"last update":"Mon May 6 13:32:02 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.47199999999999998,"15min":0.48099999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.56100000000000005,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.434,"15min":0.439},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.58699999999999997,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.34899999999999998}]},{"osd":4,"last update":"Mon May 6 13:32:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55900000000000005,"15min":0.56000000000000005},"min":{"1min":0.36199999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.623,"15min":0.59199999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":5.7149999999999999,"5min":5.7149999999999999,"15min":5.7149999999999999},"last":0.52800000000000002}]},{"osd":5,"last update":"Mon May 6 13:32:46 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.64500000000000002,"15min":0.61199999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":5.7789999999999999,"5min":5.7789999999999999,"15min":5.7789999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.65200000000000002,"15min":0.62},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":5.7480000000000002,"5min":5.7480000000000002,"15min":5.7480000000000002},"last":0.57799999999999996}]},{"osd":6,"last update":"Mon May 6 13:32:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.54800000000000004}]},{"osd":7,"last update":"Mon May 6 13:32:26 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":5.8280000000000003,"5min":5.8280000000000003,"15min":5.8280000000000003},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.52000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738420,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6924,"kb_used_data":1092,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738228,"statfs":{"total":95995035648,"available":95987945472,"internally_reserved":0,"allocated":1118208,"data_stored":769323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 6 13:32:16 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.33800000000000002},"min":{"1min":0.25700000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.373,"15min":0.35799999999999998},"min":{"1min":0.28899999999999998,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.435}]},{"osd":2,"last update":"Mon May 6 13:32:42 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.40999999999999998,"15min":0.40699999999999997},"min":{"1min":0.29699999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.54900000000000004,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.38400000000000001,"15min":0.374},"min":{"1min":0.28299999999999997,"5min":0.214,"15min":0.214},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.39100000000000001}]},{"osd":3,"last update":"Mon May 6 13:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.45900000000000002,"15min":0.46100000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.60199999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.433,"5min":0.44500000000000001,"15min":0.44700000000000001},"min":{"1min":0.32800000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.36199999999999999}]},{"osd":4,"last update":"Mon May 6 13:32:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55700000000000005,"15min":0.55400000000000005},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.57799999999999996,"15min":0.57199999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.66900000000000004}]},{"osd":5,"last update":"Mon May 6 13:32:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60399999999999998,"15min":0.60599999999999998},"min":{"1min":0.44,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.80200000000000005,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59699999999999998,"15min":0.59799999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.48999999999999999}]},{"osd":6,"last update":"Mon May 6 13:32:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.73999999999999999}]},{"osd":7,"last update":"Mon May 6 13:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.77400000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6276,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738876,"statfs":{"total":95995035648,"available":95988609024,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 13:32:39 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.35899999999999999,"15min":0.35099999999999998},"min":{"1min":0.28299999999999997,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52200000000000002,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39100000000000001,"15min":0.39300000000000002},"min":{"1min":0.251,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.47899999999999998,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.44400000000000001}]},{"osd":1,"last update":"Mon May 6 13:32:39 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.39000000000000001,"15min":0.39400000000000002},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.49399999999999999,"5min":0.622,"15min":0.622},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.40400000000000003,"15min":0.40000000000000002},"min":{"1min":0.28999999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.53900000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.36399999999999999}]},{"osd":3,"last update":"Mon May 6 13:32:00 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.432},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44800000000000001,"15min":0.44600000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.38700000000000001}]},{"osd":4,"last update":"Mon May 6 13:32:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.55600000000000005,"15min":0.55400000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.57099999999999995,"15min":0.56499999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.45700000000000002}]},{"osd":5,"last update":"Mon May 6 13:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.58299999999999996,"15min":0.57699999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78500000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.622,"15min":0.62},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.85599999999999998,"5min":1.02,"15min":1.02},"last":0.59099999999999997}]},{"osd":6,"last update":"Mon May 6 13:32:05 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.55600000000000005}]},{"osd":7,"last update":"Mon May 6 13:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.58299999999999996}]}]},{"osd":3,"up_from":26,"seq":111669149733,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 13:31:56 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.38100000000000001,"15min":0.38600000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.47399999999999998,"5min":0.502,"15min":0.502},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.41999999999999998,"15min":0.42599999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.54300000000000004,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.33200000000000002}]},{"osd":1,"last update":"Mon May 6 13:31:56 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.40100000000000002,"15min":0.40699999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.499,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.252},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.441,"15min":0.44500000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.39900000000000002}]},{"osd":2,"last update":"Mon May 6 13:31:56 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.45000000000000001,"15min":0.45300000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.60299999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.438,"15min":0.443},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57399999999999995,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.35999999999999999}]},{"osd":4,"last update":"Mon May 6 13:32:19 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.40699999999999997,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.75700000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.505},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.56100000000000005,"15min":0.55600000000000005},"min":{"1min":0.45600000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.376}]},{"osd":5,"last update":"Mon May 6 13:32:45 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.56799999999999995,"15min":0.56000000000000005},"min":{"1min":0.434,"5min":0.37,"15min":0.37},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.57299999999999995,"15min":0.56100000000000005},"min":{"1min":0.46700000000000003,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.436}]},{"osd":6,"last update":"Mon May 6 13:32:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.57099999999999995}]},{"osd":7,"last update":"Mon May 6 13:32:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.48699999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 13:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51200000000000001,"15min":0.51400000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.79500000000000004,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.58199999999999996}]},{"osd":1,"last update":"Mon May 6 13:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.54300000000000004,"15min":0.54900000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76900000000000002,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.55100000000000005,"15min":0.55500000000000005},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.84299999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.65400000000000003}]},{"osd":2,"last update":"Mon May 6 13:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.55200000000000005,"15min":0.55800000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.78300000000000003,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.54200000000000004,"15min":0.54700000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.505}]},{"osd":3,"last update":"Mon May 6 13:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.53500000000000003,"15min":0.54000000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.57599999999999996,"15min":0.58399999999999996},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.66300000000000003}]},{"osd":5,"last update":"Mon May 6 13:32:45 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.38900000000000001,"15min":0.38100000000000001},"min":{"1min":0.28799999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.41199999999999998,"15min":0.40100000000000002},"min":{"1min":0.32700000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.55500000000000005}]},{"osd":6,"last update":"Mon May 6 13:32:05 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.67400000000000004}]},{"osd":7,"last update":"Mon May 6 13:32:26 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.69199999999999995}]}]},{"osd":5,"up_from":39,"seq":167503724571,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":240072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 13:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.73899999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.67500000000000004,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.443}]},{"osd":1,"last update":"Mon May 6 13:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.747,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.002,"5min":0.63,"15min":0.56799999999999995},"min":{"1min":0.38700000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":10.869999999999999,"5min":10.869999999999999,"15min":10.869999999999999},"last":0.38600000000000001}]},{"osd":2,"last update":"Mon May 6 13:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.41599999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.72899999999999998,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.504}]},{"osd":3,"last update":"Mon May 6 13:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.56200000000000006,"15min":0.56399999999999995},"min":{"1min":0.40500000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.69199999999999995,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.41899999999999998}]},{"osd":4,"last update":"Mon May 6 13:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.32100000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.64700000000000002,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.442,"15min":0.42799999999999999},"min":{"1min":0.308,"5min":0.248,"15min":0.248},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.51500000000000001}]},{"osd":6,"last update":"Mon May 6 13:32:06 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.36599999999999999}]},{"osd":7,"last update":"Mon May 6 13:32:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.49299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T13:32:55.236 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph pg dump --format=json 2024-05-06T13:32:55.455 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T13:32:55.770 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:55 smithi077 ceph-mon[22428]: from='client.24520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-06T13:32:55.771 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:55 smithi077 ceph-mon[27166]: from='client.24520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-06T13:32:55.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:55 smithi170 ceph-mon[23272]: from='client.24520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-06T13:32:56.663 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:56 smithi077 ceph-mon[22428]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:56.664 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:56 smithi077 ceph-mon[27166]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:56.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:56 smithi170 ceph-mon[23272]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:57.051 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:32:57.052 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-05-06T13:32:57.689 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:57 smithi077 ceph-mon[22428]: from='client.24526 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-06T13:32:57.690 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:57 smithi077 ceph-mon[27166]: from='client.24526 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-06T13:32:57.717 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-05-06T13:32:56.433649+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48960,"kb_used_data":6144,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912256,"statfs":{"total":767960285184,"available":767910150144,"internally_reserved":0,"allocated":6291456,"data_stored":3510358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002101"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":266,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-05-06T13:31:44.497666+0000","last_change":"2024-05-06T13:31:30.485998+0000","last_active":"2024-05-06T13:31:44.497666+0000","last_peered":"2024-05-06T13:31:44.497666+0000","last_clean":"2024-05-06T13:31:44.497666+0000","last_became_active":"2024-05-06T13:31:30.485538+0000","last_became_peered":"2024-05-06T13:31:30.485538+0000","last_unstale":"2024-05-06T13:31:44.497666+0000","last_undegraded":"2024-05-06T13:31:44.497666+0000","last_fullsized":"2024-05-06T13:31:44.497666+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-06T13:29:33.066627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-06T13:29:33.066627+0000","last_clean_scrub_stamp":"2024-05-06T13:29:33.066627+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-07T16:18:25.066514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201300,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6096,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739056,"statfs":{"total":95995035648,"available":95988793344,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 13:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.60999999999999999}]},{"osd":1,"last update":"Mon May 6 13:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.63800000000000001}]},{"osd":2,"last update":"Mon May 6 13:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.54200000000000004}]},{"osd":3,"last update":"Mon May 6 13:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.629},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.64800000000000002}]},{"osd":4,"last update":"Mon May 6 13:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.38200000000000001}]},{"osd":5,"last update":"Mon May 6 13:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.55500000000000005}]},{"osd":6,"last update":"Mon May 6 13:32:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.50900000000000001}]}]},{"osd":6,"up_from":47,"seq":201863462937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6156,"kb_used_data":1092,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738996,"statfs":{"total":95995035648,"available":95988731904,"internally_reserved":0,"allocated":1118208,"data_stored":769265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 13:32:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.58399999999999996}]},{"osd":1,"last update":"Mon May 6 13:32:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.441},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.55100000000000005}]},{"osd":2,"last update":"Mon May 6 13:32:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":15.939,"5min":15.939,"15min":15.939},"last":0.51600000000000001}]},{"osd":3,"last update":"Mon May 6 13:32:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.499}]},{"osd":4,"last update":"Mon May 6 13:32:05 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.56399999999999995}]},{"osd":5,"last update":"Mon May 6 13:32:05 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.622},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.59299999999999997}]},{"osd":7,"last update":"Mon May 6 13:32:30 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.64200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6340,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738812,"statfs":{"total":95995035648,"available":95988543488,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 13:32:16 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.33800000000000002,"15min":0.32100000000000001},"min":{"1min":0.251,"5min":0.216,"15min":0.216},"max":{"1min":0.55800000000000005,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.37},{"interface":"front","average":{"1min":0.377,"5min":0.35599999999999998,"15min":0.32900000000000001},"min":{"1min":0.23599999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.56999999999999995,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.45600000000000002}]},{"osd":2,"last update":"Mon May 6 13:32:40 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.35599999999999998,"15min":0.35199999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.378,"5min":0.40100000000000002,"15min":0.39500000000000002},"min":{"1min":0.26100000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.35299999999999998}]},{"osd":3,"last update":"Mon May 6 13:32:02 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.47199999999999998,"15min":0.48099999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.56100000000000005,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.434,"15min":0.439},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.58699999999999997,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.32100000000000001}]},{"osd":4,"last update":"Mon May 6 13:32:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55900000000000005,"15min":0.56000000000000005},"min":{"1min":0.36199999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.623,"15min":0.59199999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":5.7149999999999999,"5min":5.7149999999999999,"15min":5.7149999999999999},"last":0.434}]},{"osd":5,"last update":"Mon May 6 13:32:46 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.64500000000000002,"15min":0.61199999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":5.7789999999999999,"5min":5.7789999999999999,"15min":5.7789999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.65200000000000002,"15min":0.62},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":5.7480000000000002,"5min":5.7480000000000002,"15min":5.7480000000000002},"last":0.55600000000000005}]},{"osd":6,"last update":"Mon May 6 13:32:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.57499999999999996}]},{"osd":7,"last update":"Mon May 6 13:32:26 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":5.8280000000000003,"5min":5.8280000000000003,"15min":5.8280000000000003},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.5}]}]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6924,"kb_used_data":1092,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738228,"statfs":{"total":95995035648,"available":95987945472,"internally_reserved":0,"allocated":1118208,"data_stored":769323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon May 6 13:32:16 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.33800000000000002},"min":{"1min":0.25700000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.373,"15min":0.35799999999999998},"min":{"1min":0.28899999999999998,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.29299999999999998}]},{"osd":2,"last update":"Mon May 6 13:32:42 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.40999999999999998,"15min":0.40699999999999997},"min":{"1min":0.29699999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.54900000000000004,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.38400000000000001,"15min":0.374},"min":{"1min":0.28299999999999997,"5min":0.214,"15min":0.214},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.31}]},{"osd":3,"last update":"Mon May 6 13:31:57 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.45900000000000002,"15min":0.46100000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.60199999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.44500000000000001,"15min":0.44700000000000001},"min":{"1min":0.32800000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.40300000000000002}]},{"osd":4,"last update":"Mon May 6 13:32:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55700000000000005,"15min":0.55400000000000005},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.57799999999999996,"15min":0.57199999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.47499999999999998}]},{"osd":5,"last update":"Mon May 6 13:32:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60399999999999998,"15min":0.60599999999999998},"min":{"1min":0.44,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.80200000000000005,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.441},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59699999999999998,"15min":0.59799999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.42499999999999999}]},{"osd":6,"last update":"Mon May 6 13:32:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.435},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.46200000000000002}]},{"osd":7,"last update":"Mon May 6 13:32:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.48499999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6276,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738876,"statfs":{"total":95995035648,"available":95988609024,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 13:32:39 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.35899999999999999,"15min":0.35099999999999998},"min":{"1min":0.28299999999999997,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52200000000000002,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.309},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39100000000000001,"15min":0.39300000000000002},"min":{"1min":0.251,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.47899999999999998,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.22}]},{"osd":1,"last update":"Mon May 6 13:32:39 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.39000000000000001,"15min":0.39400000000000002},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.49399999999999999,"5min":0.622,"15min":0.622},"last":0.254},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.40400000000000003,"15min":0.40000000000000002},"min":{"1min":0.28999999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.53900000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.26300000000000001}]},{"osd":3,"last update":"Mon May 6 13:32:00 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44800000000000001,"15min":0.44600000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.29099999999999998}]},{"osd":4,"last update":"Mon May 6 13:32:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.55600000000000005,"15min":0.55400000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.57099999999999995,"15min":0.56499999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.49099999999999999}]},{"osd":5,"last update":"Mon May 6 13:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.58299999999999996,"15min":0.57699999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78500000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.622,"15min":0.62},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.85599999999999998,"5min":1.02,"15min":1.02},"last":0.42999999999999999}]},{"osd":6,"last update":"Mon May 6 13:32:05 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.55400000000000005}]},{"osd":7,"last update":"Mon May 6 13:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.53700000000000003}]}]},{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 13:31:56 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.38100000000000001,"15min":0.38600000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.47399999999999998,"5min":0.502,"15min":0.502},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.41999999999999998,"15min":0.42599999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.54300000000000004,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.26100000000000001}]},{"osd":1,"last update":"Mon May 6 13:31:56 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.40100000000000002,"15min":0.40699999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.499,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.441,"15min":0.44500000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.32500000000000001}]},{"osd":2,"last update":"Mon May 6 13:31:56 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.45000000000000001,"15min":0.45300000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.60299999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.438,"15min":0.443},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.57399999999999995,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.31900000000000001}]},{"osd":4,"last update":"Mon May 6 13:32:19 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.40699999999999997,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.75700000000000001,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.56100000000000005,"15min":0.55600000000000005},"min":{"1min":0.45600000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.52300000000000002}]},{"osd":5,"last update":"Mon May 6 13:32:45 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.56799999999999995,"15min":0.56000000000000005},"min":{"1min":0.434,"5min":0.37,"15min":0.37},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.57299999999999995,"15min":0.56100000000000005},"min":{"1min":0.46700000000000003,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.48499999999999999}]},{"osd":6,"last update":"Mon May 6 13:32:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.502}]},{"osd":7,"last update":"Mon May 6 13:32:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.53100000000000003}]}]},{"osd":4,"up_from":31,"seq":133143986210,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 13:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51200000000000001,"15min":0.51400000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.44},{"interface":"front","average":{"1min":0.502,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.79500000000000004,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.58999999999999997}]},{"osd":1,"last update":"Mon May 6 13:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.54300000000000004,"15min":0.54900000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76900000000000002,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.55100000000000005,"15min":0.55500000000000005},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.84299999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.60199999999999998}]},{"osd":2,"last update":"Mon May 6 13:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.55200000000000005,"15min":0.55800000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.78300000000000003,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.54200000000000004,"15min":0.54700000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.55200000000000005}]},{"osd":3,"last update":"Mon May 6 13:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.53500000000000003,"15min":0.54000000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.57599999999999996,"15min":0.58399999999999996},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.56799999999999995}]},{"osd":5,"last update":"Mon May 6 13:32:45 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.38900000000000001,"15min":0.38100000000000001},"min":{"1min":0.28799999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.41199999999999998,"15min":0.40100000000000002},"min":{"1min":0.32700000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.32300000000000001}]},{"osd":6,"last update":"Mon May 6 13:32:05 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.36499999999999999}]},{"osd":7,"last update":"Mon May 6 13:32:26 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.41499999999999998}]}]},{"osd":5,"up_from":39,"seq":167503724572,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":240072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon May 6 13:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.73899999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.67500000000000004,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.61799999999999999}]},{"osd":1,"last update":"Mon May 6 13:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.747,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.002,"5min":0.63,"15min":0.56799999999999995},"min":{"1min":0.38700000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":10.869999999999999,"5min":10.869999999999999,"15min":10.869999999999999},"last":0.48799999999999999}]},{"osd":2,"last update":"Mon May 6 13:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.41599999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.72899999999999998,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.60699999999999998}]},{"osd":3,"last update":"Mon May 6 13:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.56200000000000006,"15min":0.56399999999999995},"min":{"1min":0.40500000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.69199999999999995,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.63500000000000001}]},{"osd":4,"last update":"Mon May 6 13:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.32100000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.64700000000000002,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.442,"15min":0.42799999999999999},"min":{"1min":0.308,"5min":0.248,"15min":0.248},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.54500000000000004}]},{"osd":6,"last update":"Mon May 6 13:32:06 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.65500000000000003}]},{"osd":7,"last update":"Mon May 6 13:32:28 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.66800000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-06T13:32:57.719 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-06T13:32:57.719 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-06T13:32:57.719 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-05-06T13:32:57.719 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- ceph health --format=json 2024-05-06T13:32:57.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:57 smithi170 ceph-mon[23272]: from='client.24526 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-06T13:32:57.934 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/mon.a/config 2024-05-06T13:32:58.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:32:58 smithi077 ceph-mon[22428]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:58.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:32:58 smithi077 ceph-mon[27166]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:58.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:32:58 smithi170 ceph-mon[23272]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:32:59.718 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:32:59.718 INFO:teuthology.orchestra.run.smithi077.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-06T13:33:00.362 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:00 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[45250]: level=info ts=2024-05-06T13:33:00.116Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002601785s 2024-05-06T13:33:00.387 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-05-06T13:33:00.387 INFO:tasks.cephadm:Setup complete, yielding 2024-05-06T13:33:00.387 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-06T13:33:00.397 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi077.front.sepia.ceph.com 2024-05-06T13:33:00.398 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest false --force' 2024-05-06T13:33:00.704 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:00 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/1665216156' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-06T13:33:00.704 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:00 smithi077 ceph-mon[22428]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:00.705 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:00 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/1665216156' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-06T13:33:00.705 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:00 smithi077 ceph-mon[27166]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:00.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:00 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/1665216156' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-06T13:33:00.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:00 smithi170 ceph-mon[23272]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:03.014 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-06T13:33:03.024 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi077.front.sepia.ceph.com 2024-05-06T13:33:03.024 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-05-06T13:33:03.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:03 smithi170 ceph-mon[23272]: from='client.? ' entity='client.admin' 2024-05-06T13:33:03.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:03 smithi170 ceph-mon[23272]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:03.645 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:03 smithi077 ceph-mon[22428]: from='client.? ' entity='client.admin' 2024-05-06T13:33:03.645 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:03 smithi077 ceph-mon[22428]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:03.645 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:03 smithi077 ceph-mon[27166]: from='client.? ' entity='client.admin' 2024-05-06T13:33:03.645 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:03 smithi077 ceph-mon[27166]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:04.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:33:04 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:33:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:33:04.662 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:04 smithi077 ceph-mon[22428]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:04.662 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:04 smithi077 ceph-mon[27166]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:04.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:04 smithi170 ceph-mon[23272]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:05.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:33:04 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:33:04] "GET /metrics HTTP/1.1" 200 191487 "" "Prometheus/2.33.4" 2024-05-06T13:33:06.574 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:33:06.574 INFO:teuthology.orchestra.run.smithi077.stdout: "id": "def516bf-87b9-4589-9de2-8bacd7c534aa", 2024-05-06T13:33:06.574 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "r", 2024-05-06T13:33:06.574 INFO:teuthology.orchestra.run.smithi077.stdout: "current_period": "4f1e2755-d3c5-4e4d-9a7e-f065fa840103", 2024-05-06T13:33:06.574 INFO:teuthology.orchestra.run.smithi077.stdout: "epoch": 1 2024-05-06T13:33:06.574 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:33:06.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:06 smithi170 ceph-mon[23272]: osdmap e61: 8 total, 8 up, 8 in 2024-05-06T13:33:06.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:06 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/3651800550' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-05-06T13:33:06.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:06 smithi170 ceph-mon[23272]: pgmap v45: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:06.577 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 podman[39397]: 2024-05-06 13:33:06.560159625 +0000 UTC m=+0.379479342 container start 4b872c67236dfa9268c7b071d80c951285a2fb5b2b4be664c51f1e7016339621 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8.5, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, vendor=Red Hat, Inc., release=236.1648460182, vcs-type=git, maintainer=Paul Cuzner , io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, distribution-scope=public, io.openshift.tags=base rhel8, description=Ceph Grafana Container) 2024-05-06T13:33:06.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:06 smithi077 ceph-mon[22428]: osdmap e61: 8 total, 8 up, 8 in 2024-05-06T13:33:06.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:06 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3651800550' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-05-06T13:33:06.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:06 smithi077 ceph-mon[22428]: pgmap v45: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:06.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:06 smithi077 ceph-mon[27166]: osdmap e61: 8 total, 8 up, 8 in 2024-05-06T13:33:06.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:06 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3651800550' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-05-06T13:33:06.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:06 smithi077 ceph-mon[27166]: pgmap v45: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:06.906 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-05-06T13:33:06.906 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-05-06T13:33:06.906 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-05-06T13:33:06.906 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-05-06T13:33:06.907 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-05-06T13:33:06.907 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-05-06T13:33:06.907 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-05-06T13:33:06.907 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-05-06T13:33:06.907 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-05-06T13:33:06.907 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-05-06T13:33:06.907 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-05-06T13:33:06.907 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-05-06T13:33:06.907 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-05-06T13:33:06.907 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="App mode production" logger=settings 2024-05-06T13:33:06.907 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-05-06T13:33:06.907 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-05-06T13:33:06.907 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-05-06T13:33:06.908 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-05-06T13:33:06.908 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-05-06T13:33:06.908 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-05-06T13:33:06.908 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-05-06T13:33:06.908 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-05-06T13:33:06.908 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-05-06T13:33:06.908 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-05-06T13:33:06.908 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-05-06T13:33:06.909 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-05-06T13:33:06.909 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-05-06T13:33:06.909 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-05-06T13:33:06.909 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-05-06T13:33:06.909 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-05-06T13:33:06.909 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-05-06T13:33:06.909 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-05-06T13:33:06.909 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-05-06T13:33:06.909 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-05-06T13:33:06.909 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-05-06T13:33:06.909 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-05-06T13:33:06.909 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-05-06T13:33:06.909 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-05-06T13:33:06.909 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-05-06T13:33:06.909 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-05-06T13:33:06.910 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-05-06T13:33:06.910 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-05-06T13:33:06.910 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-05-06T13:33:06.910 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-05-06T13:33:06.910 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-05-06T13:33:06.910 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-05-06T13:33:06.910 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-05-06T13:33:06.910 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-05-06T13:33:06.910 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-05-06T13:33:06.910 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-05-06T13:33:06.910 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-05-06T13:33:06.910 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-05-06T13:33:06.910 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-05-06T13:33:06.911 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-05-06T13:33:06.911 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-05-06T13:33:06.911 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-05-06T13:33:06.911 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-05-06T13:33:06.911 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-05-06T13:33:06.911 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-05-06T13:33:06.911 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-05-06T13:33:06.912 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-05-06T13:33:06.912 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-05-06T13:33:06.912 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-05-06T13:33:06.912 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-05-06T13:33:06.912 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-05-06T13:33:06.912 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-05-06T13:33:06.912 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-05-06T13:33:06.912 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-05-06T13:33:06.912 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-05-06T13:33:06.913 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-05-06T13:33:06.913 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-05-06T13:33:06.913 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-05-06T13:33:06.913 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-05-06T13:33:06.913 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-05-06T13:33:06.913 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-05-06T13:33:06.913 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-05-06T13:33:06.913 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-05-06T13:33:06.913 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-05-06T13:33:06.913 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-05-06T13:33:06.913 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-05-06T13:33:06.913 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-05-06T13:33:06.913 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-05-06T13:33:06.914 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-05-06T13:33:06.914 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-05-06T13:33:06.914 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-05-06T13:33:06.914 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-05-06T13:33:06.914 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-05-06T13:33:06.914 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-05-06T13:33:06.914 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-05-06T13:33:06.914 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-05-06T13:33:06.914 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-05-06T13:33:06.914 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-05-06T13:33:06.914 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-05-06T13:33:06.914 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-05-06T13:33:06.915 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-05-06T13:33:06.915 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-05-06T13:33:06.915 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-05-06T13:33:06.915 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-05-06T13:33:06.915 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-05-06T13:33:06.915 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-05-06T13:33:06.915 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-05-06T13:33:06.915 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-05-06T13:33:06.915 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-05-06T13:33:06.915 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-05-06T13:33:06.915 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-05-06T13:33:06.915 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-05-06T13:33:06.915 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-05-06T13:33:06.915 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-05-06T13:33:06.915 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-05-06T13:33:06.916 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-05-06T13:33:06.916 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-05-06T13:33:06.916 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-05-06T13:33:06.916 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-05-06T13:33:06.916 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-05-06T13:33:06.916 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-05-06T13:33:06.916 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-05-06T13:33:06.916 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-05-06T13:33:06.916 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-05-06T13:33:06.916 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-05-06T13:33:06.916 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-05-06T13:33:06.916 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-05-06T13:33:06.916 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-05-06T13:33:06.917 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-05-06T13:33:06.917 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-05-06T13:33:06.917 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-05-06T13:33:06.917 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-05-06T13:33:06.917 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-05-06T13:33:06.917 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-05-06T13:33:06.917 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-05-06T13:33:06.917 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-05-06T13:33:06.917 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-05-06T13:33:06.917 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-05-06T13:33:06.917 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-05-06T13:33:06.917 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-05-06T13:33:06.917 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-05-06T13:33:06.917 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-05-06T13:33:06.918 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-05-06T13:33:06.918 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-05-06T13:33:06.918 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-05-06T13:33:06.918 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-05-06T13:33:06.918 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-05-06T13:33:06.918 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-05-06T13:33:06.918 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-05-06T13:33:06.918 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-05-06T13:33:06.918 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-05-06T13:33:06.918 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-05-06T13:33:06.918 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-05-06T13:33:06.918 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-05-06T13:33:06.919 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-05-06T13:33:06.919 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-05-06T13:33:06.919 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-05-06T13:33:06.919 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-05-06T13:33:06.919 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-05-06T13:33:06.919 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-05-06T13:33:06.919 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-05-06T13:33:06.919 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-05-06T13:33:06.919 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-05-06T13:33:06.919 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-05-06T13:33:06.919 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-05-06T13:33:06.919 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-05-06T13:33:06.921 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-05-06T13:33:06.921 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-05-06T13:33:06.921 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-05-06T13:33:06.921 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-05-06T13:33:06.921 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-05-06T13:33:06.921 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-05-06T13:33:06.921 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-05-06T13:33:06.921 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-05-06T13:33:06.921 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-05-06T13:33:06.921 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-05-06T13:33:06.922 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-05-06T13:33:06.922 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-05-06T13:33:06.922 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-05-06T13:33:06.922 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-05-06T13:33:06.922 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-05-06T13:33:06.922 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-05-06T13:33:06.922 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-05-06T13:33:06.922 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-05-06T13:33:06.922 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-05-06T13:33:06.922 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-05-06T13:33:06.922 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-05-06T13:33:06.922 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-05-06T13:33:06.922 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-05-06T13:33:06.922 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-05-06T13:33:06.922 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-05-06T13:33:06.923 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-05-06T13:33:06.923 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-05-06T13:33:06.923 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-05-06T13:33:06.923 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-05-06T13:33:06.923 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-05-06T13:33:06.923 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-05-06T13:33:06.923 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-05-06T13:33:06.923 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-05-06T13:33:06.923 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-05-06T13:33:06.923 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-05-06T13:33:06.923 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-05-06T13:33:06.923 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-05-06T13:33:06.923 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-05-06T13:33:06.923 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-05-06T13:33:06.924 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-05-06T13:33:06.924 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-05-06T13:33:06.924 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-05-06T13:33:06.924 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-05-06T13:33:06.924 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-05-06T13:33:06.924 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-05-06T13:33:06.924 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-05-06T13:33:06.924 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-05-06T13:33:06.924 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-05-06T13:33:06.924 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 bash[39397]: 4b872c67236dfa9268c7b071d80c951285a2fb5b2b4be664c51f1e7016339621 2024-05-06T13:33:06.924 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-05-06T13:33:06.924 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-05-06T13:33:06.924 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-05-06T13:33:06.924 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-05-06T13:33:06.924 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-05-06T13:33:06.924 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-05-06T13:33:06.925 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-05-06T13:33:06.925 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 systemd[1]: Started Ceph grafana.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:33:06.925 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-05-06T13:33:06.925 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-05-06T13:33:06.925 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-05-06T13:33:06.925 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-05-06T13:33:06.925 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-05-06T13:33:06.925 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-05-06T13:33:06.925 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-05-06T13:33:06.925 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-05-06T13:33:06.925 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-05-06T13:33:06.925 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-05-06T13:33:06.925 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-05-06T13:33:06.925 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-05-06T13:33:06.925 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-05-06T13:33:06.925 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-05-06T13:33:06.925 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-05-06T13:33:06.926 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-05-06T13:33:06.926 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-05-06T13:33:06.926 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-05-06T13:33:06.926 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-05-06T13:33:06.926 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-05-06T13:33:06.926 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-05-06T13:33:06.926 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-05-06T13:33:06.926 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-05-06T13:33:06.926 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-05-06T13:33:06.926 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-05-06T13:33:06.926 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-05-06T13:33:06.926 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-05-06T13:33:06.926 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-05-06T13:33:06.926 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-05-06T13:33:06.926 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-05-06T13:33:06.927 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-05-06T13:33:06.927 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-05-06T13:33:06.927 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-05-06T13:33:06.927 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-05-06T13:33:06.927 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-05-06T13:33:06.927 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-05-06T13:33:06.927 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-05-06T13:33:06.927 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-05-06T13:33:06.927 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-05-06T13:33:06.927 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-05-06T13:33:06.927 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-05-06T13:33:06.927 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-05-06T13:33:06.927 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-05-06T13:33:06.927 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-05-06T13:33:06.927 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-05-06T13:33:06.927 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-05-06T13:33:06.927 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-05-06T13:33:06.928 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-05-06T13:33:06.928 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-05-06T13:33:06.928 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-05-06T13:33:06.928 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-05-06T13:33:06.928 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-05-06T13:33:06.928 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-05-06T13:33:06.928 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-05-06T13:33:06.928 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-05-06T13:33:06.928 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-05-06T13:33:06.928 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-05-06T13:33:06.928 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-05-06T13:33:06.928 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-05-06T13:33:06.928 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-05-06T13:33:06.928 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-05-06T13:33:06.928 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-05-06T13:33:06.928 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-05-06T13:33:06.929 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-05-06T13:33:06.929 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-05-06T13:33:06.929 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-05-06T13:33:06.929 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-05-06T13:33:06.929 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-05-06T13:33:06.929 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-05-06T13:33:06.929 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-05-06T13:33:06.929 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-05-06T13:33:06.929 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-05-06T13:33:06.929 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-05-06T13:33:06.929 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-05-06T13:33:06.929 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-05-06T13:33:06.929 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-05-06T13:33:06.929 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-05-06T13:33:06.929 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-05-06T13:33:06.929 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-05-06T13:33:06.930 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-05-06T13:33:06.930 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-05-06T13:33:06.930 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-05-06T13:33:06.930 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-05-06T13:33:06.930 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-05-06T13:33:06.930 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-05-06T13:33:06.930 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-05-06T13:33:06.930 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-05-06T13:33:06.930 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-05-06T13:33:06.930 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-05-06T13:33:06.930 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-05-06T13:33:06.930 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-05-06T13:33:06.930 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-05-06T13:33:06.930 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-05-06T13:33:06.930 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-05-06T13:33:06.930 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-05-06T13:33:06.931 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-05-06T13:33:06.931 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-05-06T13:33:06.931 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-05-06T13:33:06.931 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-05-06T13:33:06.931 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-05-06T13:33:06.931 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-05-06T13:33:06.931 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-05-06T13:33:06.931 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-05-06T13:33:06.931 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-05-06T13:33:06.931 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-05-06T13:33:06.931 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-05-06T13:33:06.931 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-05-06T13:33:06.931 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-05-06T13:33:06.931 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-05-06T13:33:06.931 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-05-06T13:33:06.931 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-05-06T13:33:06.932 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-05-06T13:33:06.932 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-05-06T13:33:06.932 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-05-06T13:33:06.932 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-05-06T13:33:06.932 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-05-06T13:33:06.932 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-05-06T13:33:06.932 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-05-06T13:33:06.932 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-05-06T13:33:06.932 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-05-06T13:33:06.932 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-05-06T13:33:06.932 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-05-06T13:33:06.932 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-05-06T13:33:06.932 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-05-06T13:33:06.932 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-05-06T13:33:06.932 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-05-06T13:33:06.933 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-05-06T13:33:06.933 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-05-06T13:33:06.933 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-05-06T13:33:06.933 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-05-06T13:33:06.933 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-05-06T13:33:06.933 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-05-06T13:33:06.933 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-05-06T13:33:06.933 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-05-06T13:33:06.933 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-05-06T13:33:06.933 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-05-06T13:33:06.933 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-05-06T13:33:06.933 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-05-06T13:33:06.933 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-05-06T13:33:06.933 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-05-06T13:33:06.934 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-05-06T13:33:06.934 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-05-06T13:33:06.934 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-05-06T13:33:06.934 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-05-06T13:33:06.934 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-05-06T13:33:06.934 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-05-06T13:33:06.934 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-05-06T13:33:06.934 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-05-06T13:33:06.934 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-05-06T13:33:06.934 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-05-06T13:33:06.934 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-05-06T13:33:06.934 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-05-06T13:33:06.934 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-05-06T13:33:06.934 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-05-06T13:33:06.935 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-05-06T13:33:06.935 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-05-06T13:33:06.935 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-05-06T13:33:06.935 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-05-06T13:33:06.935 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-05-06T13:33:06.935 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-05-06T13:33:06.935 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-05-06T13:33:06.935 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-05-06T13:33:06.935 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-05-06T13:33:06.935 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-05-06T13:33:06.935 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-05-06T13:33:06.935 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-05-06T13:33:06.935 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-05-06T13:33:06.935 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-05-06T13:33:06.935 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-05-06T13:33:06.935 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-05-06T13:33:06.936 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-05-06T13:33:06.936 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-05-06T13:33:06.936 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-05-06T13:33:06.936 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-05-06T13:33:06.936 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-05-06T13:33:06.936 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-05-06T13:33:06.936 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-05-06T13:33:06.936 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-05-06T13:33:06.936 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-05-06T13:33:06.937 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-05-06T13:33:06.937 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-05-06T13:33:06.937 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-05-06T13:33:06.937 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-05-06T13:33:06.937 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=170.833479ms 2024-05-06T13:33:06.937 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-05-06T13:33:06.937 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-05-06T13:33:06.937 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-05-06T13:33:06.937 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-05-06T13:33:06.937 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-05-06T13:33:06.937 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-05-06T13:33:06.937 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-05-06T13:33:06.937 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-05-06T13:33:06.937 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-05-06T13:33:06.937 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-05-06T13:33:06.937 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:33:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:33:06+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-05-06T13:33:07.220 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-05-06T13:33:07.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:07 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/3651800550' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-05-06T13:33:07.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:07 smithi170 ceph-mon[23272]: osdmap e62: 8 total, 8 up, 8 in 2024-05-06T13:33:07.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:07 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:07.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:07 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:33:07.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:07 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:33:07.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:07 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3651800550' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-05-06T13:33:07.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:07 smithi077 ceph-mon[22428]: osdmap e62: 8 total, 8 up, 8 in 2024-05-06T13:33:07.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:07 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:07.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:07 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:33:07.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:07 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:33:07.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:07 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3651800550' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-05-06T13:33:07.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:07 smithi077 ceph-mon[27166]: osdmap e62: 8 total, 8 up, 8 in 2024-05-06T13:33:07.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:07 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:07.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:07 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:33:07.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:07 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:33:08.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:08 smithi077 ceph-mon[22428]: osdmap e63: 8 total, 8 up, 8 in 2024-05-06T13:33:08.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:08 smithi077 ceph-mon[22428]: pgmap v48: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:08.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:08 smithi077 ceph-mon[27166]: osdmap e63: 8 total, 8 up, 8 in 2024-05-06T13:33:08.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:08 smithi077 ceph-mon[27166]: pgmap v48: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:08.812 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:08 smithi170 ceph-mon[23272]: osdmap e63: 8 total, 8 up, 8 in 2024-05-06T13:33:08.812 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:08 smithi170 ceph-mon[23272]: pgmap v48: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:09.576 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:33:09.577 INFO:teuthology.orchestra.run.smithi077.stdout: "id": "fc9b0745-a1ce-43af-a614-ff0f049198a7", 2024-05-06T13:33:09.577 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "default", 2024-05-06T13:33:09.577 INFO:teuthology.orchestra.run.smithi077.stdout: "api_name": "default", 2024-05-06T13:33:09.577 INFO:teuthology.orchestra.run.smithi077.stdout: "is_master": "true", 2024-05-06T13:33:09.577 INFO:teuthology.orchestra.run.smithi077.stdout: "endpoints": [], 2024-05-06T13:33:09.577 INFO:teuthology.orchestra.run.smithi077.stdout: "hostnames": [], 2024-05-06T13:33:09.577 INFO:teuthology.orchestra.run.smithi077.stdout: "hostnames_s3website": [], 2024-05-06T13:33:09.577 INFO:teuthology.orchestra.run.smithi077.stdout: "master_zone": "", 2024-05-06T13:33:09.577 INFO:teuthology.orchestra.run.smithi077.stdout: "zones": [], 2024-05-06T13:33:09.577 INFO:teuthology.orchestra.run.smithi077.stdout: "placement_targets": [], 2024-05-06T13:33:09.577 INFO:teuthology.orchestra.run.smithi077.stdout: "default_placement": "", 2024-05-06T13:33:09.577 INFO:teuthology.orchestra.run.smithi077.stdout: "realm_id": "def516bf-87b9-4589-9de2-8bacd7c534aa", 2024-05-06T13:33:09.578 INFO:teuthology.orchestra.run.smithi077.stdout: "sync_policy": { 2024-05-06T13:33:09.578 INFO:teuthology.orchestra.run.smithi077.stdout: "groups": [] 2024-05-06T13:33:09.578 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T13:33:09.578 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:33:10.164 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-05-06T13:33:10.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:10 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:10.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:10 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:10.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:10 smithi170 ceph-mon[23272]: pgmap v49: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:10.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:10 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:10.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:10 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:10.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:10 smithi077 ceph-mon[27166]: pgmap v49: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:10.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:10 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:10.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:10 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:10.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:10 smithi077 ceph-mon[22428]: pgmap v49: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:12.512 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:33:12.512 INFO:teuthology.orchestra.run.smithi077.stdout: "id": "18854a90-30b5-474a-829b-a1be1ea26e10", 2024-05-06T13:33:12.512 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "z", 2024-05-06T13:33:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "domain_root": "z.rgw.meta:root", 2024-05-06T13:33:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "control_pool": "z.rgw.control", 2024-05-06T13:33:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "gc_pool": "z.rgw.log:gc", 2024-05-06T13:33:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "lc_pool": "z.rgw.log:lc", 2024-05-06T13:33:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "log_pool": "z.rgw.log", 2024-05-06T13:33:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-05-06T13:33:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-05-06T13:33:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "roles_pool": "z.rgw.meta:roles", 2024-05-06T13:33:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-05-06T13:33:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-05-06T13:33:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-05-06T13:33:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-05-06T13:33:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-05-06T13:33:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "otp_pool": "z.rgw.otp", 2024-05-06T13:33:12.513 INFO:teuthology.orchestra.run.smithi077.stdout: "system_key": { 2024-05-06T13:33:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: "access_key": "", 2024-05-06T13:33:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: "secret_key": "" 2024-05-06T13:33:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:33:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: "placement_pools": [ 2024-05-06T13:33:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-05-06T13:33:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "default-placement", 2024-05-06T13:33:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: "val": { 2024-05-06T13:33:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: "index_pool": "z.rgw.buckets.index", 2024-05-06T13:33:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: "storage_classes": { 2024-05-06T13:33:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: "STANDARD": { 2024-05-06T13:33:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: "data_pool": "z.rgw.buckets.data" 2024-05-06T13:33:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T13:33:12.514 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:33:12.515 INFO:teuthology.orchestra.run.smithi077.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-05-06T13:33:12.515 INFO:teuthology.orchestra.run.smithi077.stdout: "index_type": 0 2024-05-06T13:33:12.515 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T13:33:12.515 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T13:33:12.515 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-05-06T13:33:12.515 INFO:teuthology.orchestra.run.smithi077.stdout: "realm_id": "def516bf-87b9-4589-9de2-8bacd7c534aa", 2024-05-06T13:33:12.515 INFO:teuthology.orchestra.run.smithi077.stdout: "notif_pool": "z.rgw.log:notif" 2024-05-06T13:33:12.515 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:33:12.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:12 smithi077 ceph-mon[22428]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 593 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-05-06T13:33:12.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:12 smithi077 ceph-mon[27166]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 593 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-05-06T13:33:12.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:12 smithi170 ceph-mon[23272]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 593 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-05-06T13:33:13.378 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-05-06T13:33:14.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:33:14 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:33:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:33:14.926 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:33:14 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:33:14] "GET /metrics HTTP/1.1" 200 193051 "" "Prometheus/2.33.4" 2024-05-06T13:33:15.204 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:14 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:15.204 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:14 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:15.204 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:14 smithi077 ceph-mon[22428]: Reconfiguring alertmanager.a (dependencies changed)... 2024-05-06T13:33:15.204 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:14 smithi077 ceph-mon[22428]: Reconfiguring daemon alertmanager.a on smithi077 2024-05-06T13:33:15.204 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:14 smithi077 ceph-mon[22428]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-05-06T13:33:15.204 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:14 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:15.204 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:14 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:15.204 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:14 smithi077 ceph-mon[27166]: Reconfiguring alertmanager.a (dependencies changed)... 2024-05-06T13:33:15.205 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:14 smithi077 ceph-mon[27166]: Reconfiguring daemon alertmanager.a on smithi077 2024-05-06T13:33:15.205 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:14 smithi077 ceph-mon[27166]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-05-06T13:33:15.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:14 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:15.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:14 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:15.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:14 smithi170 ceph-mon[23272]: Reconfiguring alertmanager.a (dependencies changed)... 2024-05-06T13:33:15.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:14 smithi170 ceph-mon[23272]: Reconfiguring daemon alertmanager.a on smithi077 2024-05-06T13:33:15.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:14 smithi170 ceph-mon[23272]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-05-06T13:33:16.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:16 smithi077 ceph-mon[22428]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 723 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-05-06T13:33:16.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:16 smithi077 ceph-mon[27166]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 723 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-05-06T13:33:16.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:16 smithi170 ceph-mon[23272]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 723 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-05-06T13:33:18.425 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:18 smithi077 systemd[1]: Stopping Ceph alertmanager.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:33:18.425 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:18 smithi077 bash[47127]: Error: no container with name or ID "ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager.a" found: no such container 2024-05-06T13:33:18.710 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:18 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[45250]: level=info ts=2024-05-06T13:33:18.425Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-05-06T13:33:18.710 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:18 smithi077 podman[47140]: 2024-05-06 13:33:18.445976598 +0000 UTC m=+0.129433773 container died 3a8b1fe0f41bee0cfe57c89c8d5dab1967c21482d4a9dcce37e659a447aaa5b0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T13:33:19.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:18 smithi077 ceph-mon[22428]: osdmap e64: 8 total, 8 up, 8 in 2024-05-06T13:33:19.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:18 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2754182431' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-05-06T13:33:19.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:18 smithi077 ceph-mon[22428]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-05-06T13:33:19.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:18 smithi077 ceph-mon[22428]: pgmap v54: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-05-06T13:33:19.052 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:18 smithi077 podman[47140]: 2024-05-06 13:33:18.810022335 +0000 UTC m=+0.493479508 container cleanup 3a8b1fe0f41bee0cfe57c89c8d5dab1967c21482d4a9dcce37e659a447aaa5b0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T13:33:19.052 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:18 smithi077 bash[47140]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a 2024-05-06T13:33:19.052 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:18 smithi077 bash[47168]: Error: no container with name or ID "ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager.a" found: no such container 2024-05-06T13:33:19.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:18 smithi077 ceph-mon[27166]: osdmap e64: 8 total, 8 up, 8 in 2024-05-06T13:33:19.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:18 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2754182431' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-05-06T13:33:19.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:18 smithi077 ceph-mon[27166]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-05-06T13:33:19.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:18 smithi077 ceph-mon[27166]: pgmap v54: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-05-06T13:33:19.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:18 smithi170 ceph-mon[23272]: osdmap e64: 8 total, 8 up, 8 in 2024-05-06T13:33:19.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:18 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2754182431' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-05-06T13:33:19.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:18 smithi170 ceph-mon[23272]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-05-06T13:33:19.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:18 smithi170 ceph-mon[23272]: pgmap v54: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-05-06T13:33:19.362 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:19 smithi077 podman[47155]: 2024-05-06 13:33:19.07796562 +0000 UTC m=+0.630805017 container remove 3a8b1fe0f41bee0cfe57c89c8d5dab1967c21482d4a9dcce37e659a447aaa5b0 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T13:33:19.362 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:19 smithi077 podman[47155]: 2024-05-06 13:33:19.153653112 +0000 UTC m=+0.706492509 volume remove c1cfeaaf5c641e7121763e7ab2aa5a7b443a3c9a66b0add808e671737af6df96 2024-05-06T13:33:19.362 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:19 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-05-06T13:33:19.362 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:19 smithi077 systemd[1]: Stopped Ceph alertmanager.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:33:19.362 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:19 smithi077 systemd[1]: Starting Ceph alertmanager.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:33:19.652 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:19 smithi077 podman[47238]: 2024-05-06 13:33:19.362896274 +0000 UTC m=+0.100453610 volume create 1b88354a20e32ecfa3a835d12b718d3f7fa5a437daa18ca512482b5161d6815f 2024-05-06T13:33:19.652 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:19 smithi077 podman[47238]: 2024-05-06 13:33:19.280656641 +0000 UTC m=+0.018213962 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-05-06T13:33:19.652 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:19 smithi077 podman[47238]: 2024-05-06 13:33:19.471710263 +0000 UTC m=+0.209267555 container create d179964192632edb46dd4058afcc7a5e82f150f1a88cf8c0272d026b1442e1d9 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T13:33:20.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:19 smithi077 ceph-mon[22428]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-05-06T13:33:20.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:19 smithi077 ceph-mon[22428]: osdmap e65: 8 total, 8 up, 8 in 2024-05-06T13:33:20.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:19 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:20.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:19 smithi077 ceph-mon[27166]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-05-06T13:33:20.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:19 smithi077 ceph-mon[27166]: osdmap e65: 8 total, 8 up, 8 in 2024-05-06T13:33:20.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:19 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:20.053 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:19 smithi077 podman[47238]: 2024-05-06 13:33:19.732934753 +0000 UTC m=+0.470492078 container init d179964192632edb46dd4058afcc7a5e82f150f1a88cf8c0272d026b1442e1d9 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T13:33:20.053 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:19 smithi077 podman[47238]: 2024-05-06 13:33:19.73562379 +0000 UTC m=+0.473181089 container start d179964192632edb46dd4058afcc7a5e82f150f1a88cf8c0272d026b1442e1d9 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T13:33:20.053 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:19 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=info ts=2024-05-06T13:33:19.745Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-05-06T13:33:20.053 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:19 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=info ts=2024-05-06T13:33:19.745Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-05-06T13:33:20.053 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:19 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=info ts=2024-05-06T13:33:19.746Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.77 port=9094 2024-05-06T13:33:20.053 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:19 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=info ts=2024-05-06T13:33:19.747Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-05-06T13:33:20.053 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:19 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=info ts=2024-05-06T13:33:19.772Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-05-06T13:33:20.053 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:19 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=info ts=2024-05-06T13:33:19.772Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-05-06T13:33:20.053 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:19 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=info ts=2024-05-06T13:33:19.774Z caller=main.go:518 msg=Listening address=:9093 2024-05-06T13:33:20.053 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:19 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=info ts=2024-05-06T13:33:19.775Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-05-06T13:33:20.053 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:19 smithi077 bash[47238]: d179964192632edb46dd4058afcc7a5e82f150f1a88cf8c0272d026b1442e1d9 2024-05-06T13:33:20.053 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:19 smithi077 systemd[1]: Started Ceph alertmanager.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:33:20.216 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:19 smithi170 ceph-mon[23272]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-05-06T13:33:20.217 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:19 smithi170 ceph-mon[23272]: osdmap e65: 8 total, 8 up, 8 in 2024-05-06T13:33:20.217 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:19 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:21.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:20 smithi077 ceph-mon[22428]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-06T13:33:21.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:20 smithi077 ceph-mon[22428]: Reconfiguring daemon prometheus.a on smithi170 2024-05-06T13:33:21.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:20 smithi077 ceph-mon[22428]: osdmap e66: 8 total, 8 up, 8 in 2024-05-06T13:33:21.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:20 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2754182431' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-05-06T13:33:21.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:20 smithi077 ceph-mon[22428]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-05-06T13:33:21.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:20 smithi077 ceph-mon[22428]: pgmap v57: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 853 B/s wr, 1 op/s 2024-05-06T13:33:21.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:20 smithi077 ceph-mon[27166]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-06T13:33:21.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:20 smithi077 ceph-mon[27166]: Reconfiguring daemon prometheus.a on smithi170 2024-05-06T13:33:21.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:20 smithi077 ceph-mon[27166]: osdmap e66: 8 total, 8 up, 8 in 2024-05-06T13:33:21.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:20 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2754182431' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-05-06T13:33:21.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:20 smithi077 ceph-mon[27166]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-05-06T13:33:21.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:20 smithi077 ceph-mon[27166]: pgmap v57: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 853 B/s wr, 1 op/s 2024-05-06T13:33:21.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:20 smithi170 ceph-mon[23272]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-06T13:33:21.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:20 smithi170 ceph-mon[23272]: Reconfiguring daemon prometheus.a on smithi170 2024-05-06T13:33:21.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:20 smithi170 ceph-mon[23272]: osdmap e66: 8 total, 8 up, 8 in 2024-05-06T13:33:21.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:20 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2754182431' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-05-06T13:33:21.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:20 smithi170 ceph-mon[23272]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-05-06T13:33:21.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:20 smithi170 ceph-mon[23272]: pgmap v57: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 853 B/s wr, 1 op/s 2024-05-06T13:33:22.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:21 smithi077 ceph-mon[27166]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-05-06T13:33:22.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:21 smithi077 ceph-mon[27166]: osdmap e67: 8 total, 8 up, 8 in 2024-05-06T13:33:22.052 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:21 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=info ts=2024-05-06T13:33:21.748Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000734779s 2024-05-06T13:33:22.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:21 smithi077 ceph-mon[22428]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-05-06T13:33:22.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:21 smithi077 ceph-mon[22428]: osdmap e67: 8 total, 8 up, 8 in 2024-05-06T13:33:22.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:21 smithi170 ceph-mon[23272]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-05-06T13:33:22.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:21 smithi170 ceph-mon[23272]: osdmap e67: 8 total, 8 up, 8 in 2024-05-06T13:33:22.940 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:22 smithi170 systemd[1]: Stopping Ceph prometheus.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:33:22.940 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:22 smithi170 bash[40088]: Error: no container with name or ID "ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus.a" found: no such container 2024-05-06T13:33:23.282 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:22 smithi170 ceph-mon[23272]: osdmap e68: 8 total, 8 up, 8 in 2024-05-06T13:33:23.282 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:22 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/4082989144' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-05-06T13:33:23.282 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:22 smithi170 ceph-mon[23272]: pgmap v60: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-05-06T13:33:23.282 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:22 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:33:22.941Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-05-06T13:33:23.282 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:22 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:33:22.941Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-05-06T13:33:23.283 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:22 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:33:22.941Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-05-06T13:33:23.283 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:22 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:33:22.941Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-05-06T13:33:23.283 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:22 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:33:22.941Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-05-06T13:33:23.283 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:22 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:33:22.941Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-05-06T13:33:23.283 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:22 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:33:22.941Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-05-06T13:33:23.283 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:22 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:33:22.941Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-05-06T13:33:23.283 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:22 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:33:22.941Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-05-06T13:33:23.283 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:22 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:33:22.941Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-05-06T13:33:23.283 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:22 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:33:22.941Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-05-06T13:33:23.283 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:22 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[38390]: ts=2024-05-06T13:33:22.941Z caller=main.go:1066 level=info msg="See you next time!" 2024-05-06T13:33:23.283 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:22 smithi170 podman[40102]: 2024-05-06 13:33:22.951336278 +0000 UTC m=+0.115433294 container died 20bf0e71db759f750d957a896d65e6a398331006eac164c041f1d0dc713ffd59 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:33:23.284 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:23 smithi170 podman[40102]: 2024-05-06 13:33:23.28279491 +0000 UTC m=+0.446891967 container cleanup 20bf0e71db759f750d957a896d65e6a398331006eac164c041f1d0dc713ffd59 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:33:23.284 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:23 smithi170 bash[40102]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a 2024-05-06T13:33:23.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:22 smithi077 ceph-mon[22428]: osdmap e68: 8 total, 8 up, 8 in 2024-05-06T13:33:23.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:22 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/4082989144' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-05-06T13:33:23.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:22 smithi077 ceph-mon[22428]: pgmap v60: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-05-06T13:33:23.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:22 smithi077 ceph-mon[27166]: osdmap e68: 8 total, 8 up, 8 in 2024-05-06T13:33:23.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:22 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/4082989144' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-05-06T13:33:23.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:22 smithi077 ceph-mon[27166]: pgmap v60: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-05-06T13:33:23.576 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:23 smithi170 podman[40116]: 2024-05-06 13:33:23.475509878 +0000 UTC m=+0.523440593 container remove 20bf0e71db759f750d957a896d65e6a398331006eac164c041f1d0dc713ffd59 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:33:23.576 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:23 smithi170 bash[40129]: Error: no container with name or ID "ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus.a" found: no such container 2024-05-06T13:33:23.576 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:23 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-05-06T13:33:23.576 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:23 smithi170 systemd[1]: Stopped Ceph prometheus.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:33:23.576 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:23 smithi170 systemd[1]: Starting Ceph prometheus.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:33:23.981 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:23 smithi170 podman[40198]: 2024-05-06 13:33:23.624415183 +0000 UTC m=+0.018639246 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-05-06T13:33:23.982 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:23 smithi170 podman[40198]: 2024-05-06 13:33:23.726571651 +0000 UTC m=+0.120795742 container create 488818ee0077edf750d1560079047234d8de5db5a9c23e04ba983c39f46b85dc (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:33:23.982 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:23 smithi170 podman[40198]: 2024-05-06 13:33:23.928600439 +0000 UTC m=+0.322824484 container init 488818ee0077edf750d1560079047234d8de5db5a9c23e04ba983c39f46b85dc (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:33:23.982 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:23 smithi170 podman[40198]: 2024-05-06 13:33:23.931128749 +0000 UTC m=+0.325352806 container start 488818ee0077edf750d1560079047234d8de5db5a9c23e04ba983c39f46b85dc (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:33:23.982 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:23 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:33:23.955Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-05-06T13:33:23.982 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:23 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:33:23.955Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-05-06T13:33:23.982 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:23 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:33:23.955Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-05-06T13:33:23.982 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:23 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:33:23.955Z caller=main.go:518 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi170 (none))" 2024-05-06T13:33:23.982 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:23 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:33:23.955Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-05-06T13:33:23.982 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:23 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:33:23.955Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-05-06T13:33:23.982 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:23 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:33:23.957Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-05-06T13:33:23.982 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:23 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:33:23.957Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-05-06T13:33:23.982 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:23 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:33:23.958Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-05-06T13:33:23.982 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:23 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:33:23.960Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-05-06T13:33:23.983 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:23 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:33:23.960Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.2µs 2024-05-06T13:33:23.983 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:23 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:33:23.960Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-05-06T13:33:24.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:23 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/4082989144' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-05-06T13:33:24.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:23 smithi077 ceph-mon[27166]: osdmap e69: 8 total, 8 up, 8 in 2024-05-06T13:33:24.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:23 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/4082989144' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-06T13:33:24.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:23 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/4082989144' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-05-06T13:33:24.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:23 smithi077 ceph-mon[27166]: osdmap e70: 8 total, 8 up, 8 in 2024-05-06T13:33:24.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:23 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/4082989144' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-05-06T13:33:24.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:23 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/4082989144' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-05-06T13:33:24.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:23 smithi077 ceph-mon[22428]: osdmap e69: 8 total, 8 up, 8 in 2024-05-06T13:33:24.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:23 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/4082989144' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-06T13:33:24.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:23 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/4082989144' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-05-06T13:33:24.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:23 smithi077 ceph-mon[22428]: osdmap e70: 8 total, 8 up, 8 in 2024-05-06T13:33:24.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:23 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/4082989144' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-05-06T13:33:24.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:23 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/4082989144' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-05-06T13:33:24.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:23 smithi170 ceph-mon[23272]: osdmap e69: 8 total, 8 up, 8 in 2024-05-06T13:33:24.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:23 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/4082989144' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-06T13:33:24.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:23 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/4082989144' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-05-06T13:33:24.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:23 smithi170 ceph-mon[23272]: osdmap e70: 8 total, 8 up, 8 in 2024-05-06T13:33:24.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:23 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/4082989144' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-05-06T13:33:24.327 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:24 smithi170 bash[40198]: 488818ee0077edf750d1560079047234d8de5db5a9c23e04ba983c39f46b85dc 2024-05-06T13:33:24.327 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:24 smithi170 systemd[1]: Started Ceph prometheus.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:33:24.327 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:33:24.165Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-05-06T13:33:24.327 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:33:24.165Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-05-06T13:33:24.327 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:33:24.165Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=40.888µs wal_replay_duration=204.535321ms total_replay_duration=204.601958ms 2024-05-06T13:33:24.327 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:33:24.168Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-05-06T13:33:24.327 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:33:24.168Z caller=main.go:947 level=info msg="TSDB started" 2024-05-06T13:33:24.327 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:33:24.168Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-05-06T13:33:24.327 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:33:24.187Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=18.807198ms db_storage=772ns remote_storage=1.709µs web_handler=817ns query_engine=727ns scrape=676.536µs scrape_sd=24.857µs notify=21.777µs notify_sd=15.32µs rules=17.718805ms 2024-05-06T13:33:24.327 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:33:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:33:24.187Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-05-06T13:33:25.012 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:33:25.012 INFO:teuthology.orchestra.run.smithi077.stdout: "id": "8b9e7df9-a363-4b08-aa6e-ec29d206df05", 2024-05-06T13:33:25.012 INFO:teuthology.orchestra.run.smithi077.stdout: "epoch": 1, 2024-05-06T13:33:25.012 INFO:teuthology.orchestra.run.smithi077.stdout: "predecessor_uuid": "4f1e2755-d3c5-4e4d-9a7e-f065fa840103", 2024-05-06T13:33:25.012 INFO:teuthology.orchestra.run.smithi077.stdout: "sync_status": [], 2024-05-06T13:33:25.012 INFO:teuthology.orchestra.run.smithi077.stdout: "period_map": { 2024-05-06T13:33:25.012 INFO:teuthology.orchestra.run.smithi077.stdout: "id": "8b9e7df9-a363-4b08-aa6e-ec29d206df05", 2024-05-06T13:33:25.012 INFO:teuthology.orchestra.run.smithi077.stdout: "zonegroups": [ 2024-05-06T13:33:25.012 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-05-06T13:33:25.012 INFO:teuthology.orchestra.run.smithi077.stdout: "id": "fc9b0745-a1ce-43af-a614-ff0f049198a7", 2024-05-06T13:33:25.012 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "default", 2024-05-06T13:33:25.012 INFO:teuthology.orchestra.run.smithi077.stdout: "api_name": "default", 2024-05-06T13:33:25.013 INFO:teuthology.orchestra.run.smithi077.stdout: "is_master": "true", 2024-05-06T13:33:25.013 INFO:teuthology.orchestra.run.smithi077.stdout: "endpoints": [], 2024-05-06T13:33:25.013 INFO:teuthology.orchestra.run.smithi077.stdout: "hostnames": [], 2024-05-06T13:33:25.013 INFO:teuthology.orchestra.run.smithi077.stdout: "hostnames_s3website": [], 2024-05-06T13:33:25.013 INFO:teuthology.orchestra.run.smithi077.stdout: "master_zone": "18854a90-30b5-474a-829b-a1be1ea26e10", 2024-05-06T13:33:25.013 INFO:teuthology.orchestra.run.smithi077.stdout: "zones": [ 2024-05-06T13:33:25.013 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-05-06T13:33:25.013 INFO:teuthology.orchestra.run.smithi077.stdout: "id": "18854a90-30b5-474a-829b-a1be1ea26e10", 2024-05-06T13:33:25.013 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "z", 2024-05-06T13:33:25.013 INFO:teuthology.orchestra.run.smithi077.stdout: "endpoints": [], 2024-05-06T13:33:25.013 INFO:teuthology.orchestra.run.smithi077.stdout: "log_meta": "false", 2024-05-06T13:33:25.013 INFO:teuthology.orchestra.run.smithi077.stdout: "log_data": "false", 2024-05-06T13:33:25.013 INFO:teuthology.orchestra.run.smithi077.stdout: "bucket_index_max_shards": 11, 2024-05-06T13:33:25.013 INFO:teuthology.orchestra.run.smithi077.stdout: "read_only": "false", 2024-05-06T13:33:25.014 INFO:teuthology.orchestra.run.smithi077.stdout: "tier_type": "", 2024-05-06T13:33:25.014 INFO:teuthology.orchestra.run.smithi077.stdout: "sync_from_all": "true", 2024-05-06T13:33:25.014 INFO:teuthology.orchestra.run.smithi077.stdout: "sync_from": [], 2024-05-06T13:33:25.014 INFO:teuthology.orchestra.run.smithi077.stdout: "redirect_zone": "" 2024-05-06T13:33:25.014 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T13:33:25.014 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-05-06T13:33:25.014 INFO:teuthology.orchestra.run.smithi077.stdout: "placement_targets": [ 2024-05-06T13:33:25.014 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-05-06T13:33:25.014 INFO:teuthology.orchestra.run.smithi077.stdout: "name": "default-placement", 2024-05-06T13:33:25.014 INFO:teuthology.orchestra.run.smithi077.stdout: "tags": [], 2024-05-06T13:33:25.014 INFO:teuthology.orchestra.run.smithi077.stdout: "storage_classes": [ 2024-05-06T13:33:25.014 INFO:teuthology.orchestra.run.smithi077.stdout: "STANDARD" 2024-05-06T13:33:25.014 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-05-06T13:33:25.014 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T13:33:25.015 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-05-06T13:33:25.015 INFO:teuthology.orchestra.run.smithi077.stdout: "default_placement": "default-placement", 2024-05-06T13:33:25.015 INFO:teuthology.orchestra.run.smithi077.stdout: "realm_id": "def516bf-87b9-4589-9de2-8bacd7c534aa", 2024-05-06T13:33:25.015 INFO:teuthology.orchestra.run.smithi077.stdout: "sync_policy": { 2024-05-06T13:33:25.015 INFO:teuthology.orchestra.run.smithi077.stdout: "groups": [] 2024-05-06T13:33:25.015 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T13:33:25.015 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T13:33:25.015 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-05-06T13:33:25.015 INFO:teuthology.orchestra.run.smithi077.stdout: "short_zone_ids": [ 2024-05-06T13:33:25.015 INFO:teuthology.orchestra.run.smithi077.stdout: { 2024-05-06T13:33:25.015 INFO:teuthology.orchestra.run.smithi077.stdout: "key": "18854a90-30b5-474a-829b-a1be1ea26e10", 2024-05-06T13:33:25.015 INFO:teuthology.orchestra.run.smithi077.stdout: "val": 1052097718 2024-05-06T13:33:25.015 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T13:33:25.015 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-05-06T13:33:25.015 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:33:25.016 INFO:teuthology.orchestra.run.smithi077.stdout: "master_zonegroup": "fc9b0745-a1ce-43af-a614-ff0f049198a7", 2024-05-06T13:33:25.016 INFO:teuthology.orchestra.run.smithi077.stdout: "master_zone": "18854a90-30b5-474a-829b-a1be1ea26e10", 2024-05-06T13:33:25.016 INFO:teuthology.orchestra.run.smithi077.stdout: "period_config": { 2024-05-06T13:33:25.016 INFO:teuthology.orchestra.run.smithi077.stdout: "bucket_quota": { 2024-05-06T13:33:25.016 INFO:teuthology.orchestra.run.smithi077.stdout: "enabled": false, 2024-05-06T13:33:25.016 INFO:teuthology.orchestra.run.smithi077.stdout: "check_on_raw": false, 2024-05-06T13:33:25.016 INFO:teuthology.orchestra.run.smithi077.stdout: "max_size": -1, 2024-05-06T13:33:25.016 INFO:teuthology.orchestra.run.smithi077.stdout: "max_size_kb": 0, 2024-05-06T13:33:25.016 INFO:teuthology.orchestra.run.smithi077.stdout: "max_objects": -1 2024-05-06T13:33:25.016 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:33:25.016 INFO:teuthology.orchestra.run.smithi077.stdout: "user_quota": { 2024-05-06T13:33:25.016 INFO:teuthology.orchestra.run.smithi077.stdout: "enabled": false, 2024-05-06T13:33:25.016 INFO:teuthology.orchestra.run.smithi077.stdout: "check_on_raw": false, 2024-05-06T13:33:25.016 INFO:teuthology.orchestra.run.smithi077.stdout: "max_size": -1, 2024-05-06T13:33:25.016 INFO:teuthology.orchestra.run.smithi077.stdout: "max_size_kb": 0, 2024-05-06T13:33:25.017 INFO:teuthology.orchestra.run.smithi077.stdout: "max_objects": -1 2024-05-06T13:33:25.017 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:33:25.017 INFO:teuthology.orchestra.run.smithi077.stdout: "user_ratelimit": { 2024-05-06T13:33:25.017 INFO:teuthology.orchestra.run.smithi077.stdout: "max_read_ops": 0, 2024-05-06T13:33:25.017 INFO:teuthology.orchestra.run.smithi077.stdout: "max_write_ops": 0, 2024-05-06T13:33:25.017 INFO:teuthology.orchestra.run.smithi077.stdout: "max_read_bytes": 0, 2024-05-06T13:33:25.017 INFO:teuthology.orchestra.run.smithi077.stdout: "max_write_bytes": 0, 2024-05-06T13:33:25.017 INFO:teuthology.orchestra.run.smithi077.stdout: "enabled": false 2024-05-06T13:33:25.017 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:33:25.017 INFO:teuthology.orchestra.run.smithi077.stdout: "bucket_ratelimit": { 2024-05-06T13:33:25.017 INFO:teuthology.orchestra.run.smithi077.stdout: "max_read_ops": 0, 2024-05-06T13:33:25.017 INFO:teuthology.orchestra.run.smithi077.stdout: "max_write_ops": 0, 2024-05-06T13:33:25.017 INFO:teuthology.orchestra.run.smithi077.stdout: "max_read_bytes": 0, 2024-05-06T13:33:25.017 INFO:teuthology.orchestra.run.smithi077.stdout: "max_write_bytes": 0, 2024-05-06T13:33:25.017 INFO:teuthology.orchestra.run.smithi077.stdout: "enabled": false 2024-05-06T13:33:25.018 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:33:25.018 INFO:teuthology.orchestra.run.smithi077.stdout: "anonymous_ratelimit": { 2024-05-06T13:33:25.018 INFO:teuthology.orchestra.run.smithi077.stdout: "max_read_ops": 0, 2024-05-06T13:33:25.018 INFO:teuthology.orchestra.run.smithi077.stdout: "max_write_ops": 0, 2024-05-06T13:33:25.018 INFO:teuthology.orchestra.run.smithi077.stdout: "max_read_bytes": 0, 2024-05-06T13:33:25.018 INFO:teuthology.orchestra.run.smithi077.stdout: "max_write_bytes": 0, 2024-05-06T13:33:25.018 INFO:teuthology.orchestra.run.smithi077.stdout: "enabled": false 2024-05-06T13:33:25.018 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T13:33:25.018 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:33:25.018 INFO:teuthology.orchestra.run.smithi077.stdout: "realm_id": "def516bf-87b9-4589-9de2-8bacd7c534aa", 2024-05-06T13:33:25.018 INFO:teuthology.orchestra.run.smithi077.stdout: "realm_name": "r", 2024-05-06T13:33:25.018 INFO:teuthology.orchestra.run.smithi077.stdout: "realm_epoch": 2 2024-05-06T13:33:25.018 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:33:25.292 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:25.292 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T13:33:25.292 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[22428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T13:33:25.293 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.77:9093"}]: dispatch 2024-05-06T13:33:25.293 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[22428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.77:9093"}]: dispatch 2024-05-06T13:33:25.293 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:25.293 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:33:25.293 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[22428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:33:25.293 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.170:9095"}]: dispatch 2024-05-06T13:33:25.293 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[22428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.170:9095"}]: dispatch 2024-05-06T13:33:25.293 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:25.293 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T13:33:25.293 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[22428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T13:33:25.293 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.170:3000"}]: dispatch 2024-05-06T13:33:25.293 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[22428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.170:3000"}]: dispatch 2024-05-06T13:33:25.293 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:25.293 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:33:25.293 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:33:25.294 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[22428]: pgmap v63: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-05-06T13:33:25.294 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/4082989144' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-05-06T13:33:25.294 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[22428]: osdmap e71: 8 total, 8 up, 8 in 2024-05-06T13:33:25.294 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:25.294 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T13:33:25.294 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[27166]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T13:33:25.294 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.77:9093"}]: dispatch 2024-05-06T13:33:25.294 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[27166]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.77:9093"}]: dispatch 2024-05-06T13:33:25.294 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:25.294 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:33:25.294 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[27166]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:33:25.294 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.170:9095"}]: dispatch 2024-05-06T13:33:25.294 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[27166]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.170:9095"}]: dispatch 2024-05-06T13:33:25.295 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:25.295 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T13:33:25.295 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[27166]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T13:33:25.295 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.170:3000"}]: dispatch 2024-05-06T13:33:25.295 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[27166]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.170:3000"}]: dispatch 2024-05-06T13:33:25.295 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:25.295 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:33:25.295 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:33:25.295 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[27166]: pgmap v63: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-05-06T13:33:25.295 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/4082989144' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-05-06T13:33:25.295 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:25 smithi077 ceph-mon[27166]: osdmap e71: 8 total, 8 up, 8 in 2024-05-06T13:33:25.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:25 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:25.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:25 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T13:33:25.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:25 smithi170 ceph-mon[23272]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T13:33:25.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:25 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.77:9093"}]: dispatch 2024-05-06T13:33:25.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:25 smithi170 ceph-mon[23272]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.77:9093"}]: dispatch 2024-05-06T13:33:25.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:25 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:25.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:25 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:33:25.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:25 smithi170 ceph-mon[23272]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:33:25.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:25 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.170:9095"}]: dispatch 2024-05-06T13:33:25.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:25 smithi170 ceph-mon[23272]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.170:9095"}]: dispatch 2024-05-06T13:33:25.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:25 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:25.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:25 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T13:33:25.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:25 smithi170 ceph-mon[23272]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T13:33:25.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:25 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.170:3000"}]: dispatch 2024-05-06T13:33:25.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:25 smithi170 ceph-mon[23272]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.170:3000"}]: dispatch 2024-05-06T13:33:25.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:25 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:25.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:25 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:33:25.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:25 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:33:25.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:25 smithi170 ceph-mon[23272]: pgmap v63: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-05-06T13:33:25.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:25 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/4082989144' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-05-06T13:33:25.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:25 smithi170 ceph-mon[23272]: osdmap e71: 8 total, 8 up, 8 in 2024-05-06T13:33:25.927 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-05-06T13:33:26.795 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:26 smithi077 ceph-mon[22428]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-05-06T13:33:26.795 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:26 smithi077 ceph-mon[27166]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-05-06T13:33:26.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:26 smithi170 ceph-mon[23272]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-05-06T13:33:28.786 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:28 smithi077 ceph-mon[22428]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-05-06T13:33:28.786 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:28 smithi077 ceph-mon[27166]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-05-06T13:33:28.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:28 smithi170 ceph-mon[23272]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-05-06T13:33:29.203 INFO:teuthology.orchestra.run.smithi077.stdout:Scheduled rgw.foo update... 2024-05-06T13:33:29.798 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:29 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=info ts=2024-05-06T13:33:29.750Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003155381s 2024-05-06T13:33:29.834 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph osd pool create foo' 2024-05-06T13:33:30.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:29 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:30.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:29 smithi077 ceph-mon[22428]: from='client.24601 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:33:30.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:29 smithi077 ceph-mon[22428]: Saving service rgw.foo spec with placement count:2 2024-05-06T13:33:30.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:29 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:30.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:29 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:30.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:29 smithi077 ceph-mon[27166]: from='client.24601 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:33:30.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:29 smithi077 ceph-mon[27166]: Saving service rgw.foo spec with placement count:2 2024-05-06T13:33:30.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:29 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:30.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:29 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:30.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:29 smithi170 ceph-mon[23272]: from='client.24601 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:33:30.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:29 smithi170 ceph-mon[23272]: Saving service rgw.foo spec with placement count:2 2024-05-06T13:33:30.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:29 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:31.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:30 smithi170 ceph-mon[23272]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-05-06T13:33:31.166 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:30 smithi077 ceph-mon[22428]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-05-06T13:33:31.166 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:30 smithi077 ceph-mon[27166]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-05-06T13:33:32.084 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'foo' created 2024-05-06T13:33:32.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:32 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/461722109' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-05-06T13:33:32.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:32 smithi170 ceph-mon[23272]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-05-06T13:33:32.426 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:32 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/461722109' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-05-06T13:33:32.426 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:32 smithi077 ceph-mon[22428]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-05-06T13:33:32.426 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:32 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/461722109' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-05-06T13:33:32.426 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:32 smithi077 ceph-mon[27166]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-05-06T13:33:32.800 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'rbd pool init foo' 2024-05-06T13:33:33.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:33 smithi077 ceph-mon[27166]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-05-06T13:33:33.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:33 smithi077 ceph-mon[27166]: osdmap e72: 8 total, 8 up, 8 in 2024-05-06T13:33:33.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:33 smithi077 ceph-mon[27166]: pgmap v69: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-05-06T13:33:33.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:33 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:33.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:33 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:33.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:33 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:33.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:33 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:33.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:33 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:33.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:33 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi077.xctpgy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T13:33:33.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:33 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi077.xctpgy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-06T13:33:33.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:33 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:33.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:33 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:33:33.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:33 smithi077 ceph-mon[22428]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-05-06T13:33:33.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:33 smithi077 ceph-mon[22428]: osdmap e72: 8 total, 8 up, 8 in 2024-05-06T13:33:33.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:33 smithi077 ceph-mon[22428]: pgmap v69: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-05-06T13:33:33.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:33 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:33.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:33 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:33.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:33 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:33.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:33 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:33.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:33 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:33.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:33 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi077.xctpgy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T13:33:33.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:33 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi077.xctpgy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-06T13:33:33.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:33 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:33.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:33 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:33:33.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:33 smithi170 ceph-mon[23272]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-05-06T13:33:33.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:33 smithi170 ceph-mon[23272]: osdmap e72: 8 total, 8 up, 8 in 2024-05-06T13:33:33.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:33 smithi170 ceph-mon[23272]: pgmap v69: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-05-06T13:33:33.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:33 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:33.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:33 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:33.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:33 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:33.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:33 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:33.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:33 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:33.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:33 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi077.xctpgy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T13:33:33.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:33 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi077.xctpgy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-06T13:33:33.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:33 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:33.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:33 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:33:34.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:34 smithi170 ceph-mon[23272]: Saving service rgw.foo spec with placement count:2 2024-05-06T13:33:34.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:34 smithi170 ceph-mon[23272]: Deploying daemon rgw.foo.smithi077.xctpgy on smithi077 2024-05-06T13:33:34.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:34 smithi170 ceph-mon[23272]: osdmap e73: 8 total, 8 up, 8 in 2024-05-06T13:33:34.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:34 smithi170 ceph-mon[23272]: osdmap e74: 8 total, 8 up, 8 in 2024-05-06T13:33:34.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:33:34 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:33:34] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:33:34.374 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:34 smithi077 ceph-mon[22428]: Saving service rgw.foo spec with placement count:2 2024-05-06T13:33:34.374 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:34 smithi077 ceph-mon[22428]: Deploying daemon rgw.foo.smithi077.xctpgy on smithi077 2024-05-06T13:33:34.374 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:34 smithi077 ceph-mon[22428]: osdmap e73: 8 total, 8 up, 8 in 2024-05-06T13:33:34.374 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:34 smithi077 ceph-mon[22428]: osdmap e74: 8 total, 8 up, 8 in 2024-05-06T13:33:34.375 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:34 smithi077 ceph-mon[27166]: Saving service rgw.foo spec with placement count:2 2024-05-06T13:33:34.375 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:34 smithi077 ceph-mon[27166]: Deploying daemon rgw.foo.smithi077.xctpgy on smithi077 2024-05-06T13:33:34.375 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:34 smithi077 ceph-mon[27166]: osdmap e73: 8 total, 8 up, 8 in 2024-05-06T13:33:34.375 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:34 smithi077 ceph-mon[27166]: osdmap e74: 8 total, 8 up, 8 in 2024-05-06T13:33:35.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:33:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:33:34] "GET /metrics HTTP/1.1" 200 196867 "" "Prometheus/2.33.4" 2024-05-06T13:33:35.354 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:35 smithi077 ceph-mon[22428]: pgmap v72: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:35.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:35 smithi077 ceph-mon[27166]: pgmap v72: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:35.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:35 smithi170 ceph-mon[23272]: pgmap v72: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:36.199 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:36 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/1341849776' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-05-06T13:33:36.199 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:36 smithi077 ceph-mon[22428]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-05-06T13:33:36.200 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:36 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/1341849776' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-05-06T13:33:36.200 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:36 smithi077 ceph-mon[27166]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-05-06T13:33:36.529 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:36 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/1341849776' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-05-06T13:33:36.529 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:36 smithi170 ceph-mon[23272]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-05-06T13:33:37.382 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:37 smithi077 ceph-mon[22428]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-05-06T13:33:37.382 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:37 smithi077 ceph-mon[22428]: osdmap e75: 8 total, 8 up, 8 in 2024-05-06T13:33:37.382 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:37 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:37.382 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:37 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi170.rackmk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T13:33:37.382 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:37 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi170.rackmk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-06T13:33:37.382 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:37 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:37.382 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:37 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:33:37.383 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:37 smithi077 ceph-mon[22428]: Deploying daemon rgw.foo.smithi170.rackmk on smithi170 2024-05-06T13:33:37.383 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:37 smithi077 ceph-mon[22428]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:37.383 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:37 smithi077 ceph-mon[27166]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-05-06T13:33:37.383 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:37 smithi077 ceph-mon[27166]: osdmap e75: 8 total, 8 up, 8 in 2024-05-06T13:33:37.383 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:37 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:37.383 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:37 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi170.rackmk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T13:33:37.383 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:37 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi170.rackmk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-06T13:33:37.383 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:37 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:37.383 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:37 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:33:37.383 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:37 smithi077 ceph-mon[27166]: Deploying daemon rgw.foo.smithi170.rackmk on smithi170 2024-05-06T13:33:37.384 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:37 smithi077 ceph-mon[27166]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:37.467 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:37 smithi170 ceph-mon[23272]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-05-06T13:33:37.467 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:37 smithi170 ceph-mon[23272]: osdmap e75: 8 total, 8 up, 8 in 2024-05-06T13:33:37.468 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:37 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:37.468 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:37 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi170.rackmk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T13:33:37.468 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:37 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi170.rackmk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-06T13:33:37.468 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:37 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:37.468 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:37 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:33:37.468 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:37 smithi170 ceph-mon[23272]: Deploying daemon rgw.foo.smithi170.rackmk on smithi170 2024-05-06T13:33:37.468 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:37 smithi170 ceph-mon[23272]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:38.434 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:38 smithi170 ceph-mon[23272]: osdmap e76: 8 total, 8 up, 8 in 2024-05-06T13:33:38.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:38 smithi077 ceph-mon[22428]: osdmap e76: 8 total, 8 up, 8 in 2024-05-06T13:33:38.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:38 smithi077 ceph-mon[27166]: osdmap e76: 8 total, 8 up, 8 in 2024-05-06T13:33:39.039 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch apply iscsi foo u p' 2024-05-06T13:33:39.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:39 smithi077 ceph-mon[22428]: osdmap e77: 8 total, 8 up, 8 in 2024-05-06T13:33:39.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:39 smithi077 ceph-mon[22428]: pgmap v77: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:39.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:39 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:39.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:39 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:33:39.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:39 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:33:39.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:39 smithi077 ceph-mon[27166]: osdmap e77: 8 total, 8 up, 8 in 2024-05-06T13:33:39.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:39 smithi077 ceph-mon[27166]: pgmap v77: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:39.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:39 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:39.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:39 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:33:39.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:39 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:33:39.471 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:39 smithi170 ceph-mon[23272]: osdmap e77: 8 total, 8 up, 8 in 2024-05-06T13:33:39.472 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:39 smithi170 ceph-mon[23272]: pgmap v77: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:39.472 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:39 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:39.472 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:39 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:33:39.472 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:39 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:33:40.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:40 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:40.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:40 smithi170 ceph-mon[23272]: pgmap v78: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:40.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:40 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:40.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:40 smithi077 ceph-mon[22428]: pgmap v78: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:40.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:40 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:40.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:40 smithi077 ceph-mon[27166]: pgmap v78: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:33:41.747 INFO:teuthology.orchestra.run.smithi077.stdout:Scheduled iscsi.foo update... 2024-05-06T13:33:42.919 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:42 smithi077 ceph-mon[22428]: from='client.24658 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:33:42.919 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:42 smithi077 ceph-mon[22428]: Saving service iscsi.foo spec with placement count:1 2024-05-06T13:33:42.919 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:42 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:42.919 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:42 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:42.919 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:42 smithi077 ceph-mon[22428]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 176 KiB/s rd, 5.7 KiB/s wr, 335 op/s 2024-05-06T13:33:42.920 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:42 smithi077 ceph-mon[27166]: from='client.24658 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:33:42.920 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:42 smithi077 ceph-mon[27166]: Saving service iscsi.foo spec with placement count:1 2024-05-06T13:33:42.920 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:42 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:42.920 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:42 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:42.920 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:42 smithi077 ceph-mon[27166]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 176 KiB/s rd, 5.7 KiB/s wr, 335 op/s 2024-05-06T13:33:42.945 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'sleep 180' 2024-05-06T13:33:43.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:42 smithi170 ceph-mon[23272]: from='client.24658 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:33:43.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:42 smithi170 ceph-mon[23272]: Saving service iscsi.foo spec with placement count:1 2024-05-06T13:33:43.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:42 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:43.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:42 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:43.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:42 smithi170 ceph-mon[23272]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 176 KiB/s rd, 5.7 KiB/s wr, 335 op/s 2024-05-06T13:33:44.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:33:44 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:33:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:33:44.802 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:33:44 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:33:44] "GET /metrics HTTP/1.1" 200 207956 "" "Prometheus/2.33.4" 2024-05-06T13:33:44.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:44.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[22428]: Checking dashboard <-> RGW credentials 2024-05-06T13:33:44.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:44.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:44.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:44.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[22428]: Checking pool "foo" exists for service iscsi.foo 2024-05-06T13:33:44.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:33:44.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-05-06T13:33:44.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:33:44.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[22428]: Deploying daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:33:44.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-05-06T13:33:44.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-05-06T13:33:44.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-05-06T13:33:44.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-05-06T13:33:44.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[22428]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 4.5 KiB/s wr, 265 op/s 2024-05-06T13:33:44.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:33:44.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:33:44.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]': finished 2024-05-06T13:33:44.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-05-06T13:33:44.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-05-06T13:33:44.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-05-06T13:33:44.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[22428]: osdmap e78: 8 total, 8 up, 8 in 2024-05-06T13:33:44.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:44.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[27166]: Checking dashboard <-> RGW credentials 2024-05-06T13:33:44.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:44.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:44.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:44.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[27166]: Checking pool "foo" exists for service iscsi.foo 2024-05-06T13:33:44.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:33:44.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-05-06T13:33:44.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:33:44.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[27166]: Deploying daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:33:44.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-05-06T13:33:44.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-05-06T13:33:44.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-05-06T13:33:44.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-05-06T13:33:44.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[27166]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 4.5 KiB/s wr, 265 op/s 2024-05-06T13:33:44.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:33:44.805 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:33:44.805 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]': finished 2024-05-06T13:33:44.805 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-05-06T13:33:44.805 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-05-06T13:33:44.805 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-05-06T13:33:44.805 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:44 smithi077 ceph-mon[27166]: osdmap e78: 8 total, 8 up, 8 in 2024-05-06T13:33:45.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:45.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:44 smithi170 ceph-mon[23272]: Checking dashboard <-> RGW credentials 2024-05-06T13:33:45.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:45.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:45.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:45.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:44 smithi170 ceph-mon[23272]: Checking pool "foo" exists for service iscsi.foo 2024-05-06T13:33:45.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:33:45.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-05-06T13:33:45.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:33:45.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:44 smithi170 ceph-mon[23272]: Deploying daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:33:45.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-05-06T13:33:45.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-05-06T13:33:45.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-05-06T13:33:45.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-05-06T13:33:45.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:44 smithi170 ceph-mon[23272]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 4.5 KiB/s wr, 265 op/s 2024-05-06T13:33:45.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:33:45.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:33:45.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]': finished 2024-05-06T13:33:45.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-05-06T13:33:45.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-05-06T13:33:45.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-05-06T13:33:45.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:44 smithi170 ceph-mon[23272]: osdmap e78: 8 total, 8 up, 8 in 2024-05-06T13:33:46.050 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:45 smithi077 ceph-mon[22428]: mgrmap e20: y(active, since 2m), standbys: x 2024-05-06T13:33:46.050 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:45 smithi077 ceph-mon[22428]: osdmap e79: 8 total, 8 up, 8 in 2024-05-06T13:33:46.050 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:45 smithi077 ceph-mon[27166]: mgrmap e20: y(active, since 2m), standbys: x 2024-05-06T13:33:46.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:45 smithi077 ceph-mon[27166]: osdmap e79: 8 total, 8 up, 8 in 2024-05-06T13:33:46.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:45 smithi170 ceph-mon[23272]: mgrmap e20: y(active, since 2m), standbys: x 2024-05-06T13:33:46.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:45 smithi170 ceph-mon[23272]: osdmap e79: 8 total, 8 up, 8 in 2024-05-06T13:33:47.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:46 smithi077 ceph-mon[22428]: pgmap v83: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s rd, 4.7 KiB/s wr, 270 op/s 2024-05-06T13:33:47.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:46 smithi077 ceph-mon[27166]: pgmap v83: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s rd, 4.7 KiB/s wr, 270 op/s 2024-05-06T13:33:47.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:46 smithi170 ceph-mon[23272]: pgmap v83: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s rd, 4.7 KiB/s wr, 270 op/s 2024-05-06T13:33:49.007 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:48 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:49.037 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:48 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:33:49.037 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:48 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:33:49.037 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:48 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/692481206' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:33:49.037 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:48 smithi077 ceph-mon[22428]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s rd, 4.7 KiB/s wr, 270 op/s 2024-05-06T13:33:49.037 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:48 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/131242088' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1194062530"}]: dispatch 2024-05-06T13:33:49.037 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:48 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:49.037 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:48 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:33:49.038 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:48 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:33:49.038 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:48 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/692481206' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:33:49.038 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:48 smithi077 ceph-mon[27166]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s rd, 4.7 KiB/s wr, 270 op/s 2024-05-06T13:33:49.038 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:48 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/131242088' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1194062530"}]: dispatch 2024-05-06T13:33:49.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:48 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:49.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:48 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:33:49.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:48 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:33:49.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:48 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/692481206' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:33:49.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:48 smithi170 ceph-mon[23272]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s rd, 4.7 KiB/s wr, 270 op/s 2024-05-06T13:33:49.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:48 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/131242088' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1194062530"}]: dispatch 2024-05-06T13:33:50.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:49 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/131242088' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1194062530"}]': finished 2024-05-06T13:33:50.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:49 smithi077 ceph-mon[22428]: osdmap e80: 8 total, 8 up, 8 in 2024-05-06T13:33:50.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:49 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/4070508842' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/3607570357"}]: dispatch 2024-05-06T13:33:50.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:49 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/3607570357"}]: dispatch 2024-05-06T13:33:50.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:49 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:50.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:49 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/131242088' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1194062530"}]': finished 2024-05-06T13:33:50.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:49 smithi077 ceph-mon[27166]: osdmap e80: 8 total, 8 up, 8 in 2024-05-06T13:33:50.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:49 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/4070508842' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/3607570357"}]: dispatch 2024-05-06T13:33:50.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:49 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/3607570357"}]: dispatch 2024-05-06T13:33:50.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:49 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:50.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:49 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/131242088' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1194062530"}]': finished 2024-05-06T13:33:50.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:49 smithi170 ceph-mon[23272]: osdmap e80: 8 total, 8 up, 8 in 2024-05-06T13:33:50.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:49 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/4070508842' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/3607570357"}]: dispatch 2024-05-06T13:33:50.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:49 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/3607570357"}]: dispatch 2024-05-06T13:33:50.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:49 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:51.255 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:51 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/3607570357"}]': finished 2024-05-06T13:33:51.256 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:51 smithi077 ceph-mon[22428]: osdmap e81: 8 total, 8 up, 8 in 2024-05-06T13:33:51.256 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:51 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3219199580' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/3988065694"}]: dispatch 2024-05-06T13:33:51.256 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:51 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/3988065694"}]: dispatch 2024-05-06T13:33:51.256 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:51 smithi077 ceph-mon[22428]: pgmap v87: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 344 B/s wr, 7 op/s 2024-05-06T13:33:51.256 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:51 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/3607570357"}]': finished 2024-05-06T13:33:51.256 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:51 smithi077 ceph-mon[27166]: osdmap e81: 8 total, 8 up, 8 in 2024-05-06T13:33:51.256 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:51 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3219199580' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/3988065694"}]: dispatch 2024-05-06T13:33:51.256 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:51 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/3988065694"}]: dispatch 2024-05-06T13:33:51.256 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:51 smithi077 ceph-mon[27166]: pgmap v87: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 344 B/s wr, 7 op/s 2024-05-06T13:33:51.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:50 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/3607570357"}]': finished 2024-05-06T13:33:51.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:50 smithi170 ceph-mon[23272]: osdmap e81: 8 total, 8 up, 8 in 2024-05-06T13:33:51.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:50 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/3219199580' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/3988065694"}]: dispatch 2024-05-06T13:33:51.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:50 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/3988065694"}]: dispatch 2024-05-06T13:33:51.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:50 smithi170 ceph-mon[23272]: pgmap v87: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 344 B/s wr, 7 op/s 2024-05-06T13:33:52.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/3988065694"}]': finished 2024-05-06T13:33:52.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[22428]: osdmap e82: 8 total, 8 up, 8 in 2024-05-06T13:33:52.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:52.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/1424213713' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/371454062"}]: dispatch 2024-05-06T13:33:52.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/371454062"}]: dispatch 2024-05-06T13:33:52.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:52.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:52.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:33:52.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:33:52.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:52.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi077"}]: dispatch 2024-05-06T13:33:52.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:52.305 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:33:52.305 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:33:52.305 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:52.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/3988065694"}]': finished 2024-05-06T13:33:52.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[27166]: osdmap e82: 8 total, 8 up, 8 in 2024-05-06T13:33:52.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:52.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/1424213713' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/371454062"}]: dispatch 2024-05-06T13:33:52.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/371454062"}]: dispatch 2024-05-06T13:33:52.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:52.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:52.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:33:52.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:33:52.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:52.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi077"}]: dispatch 2024-05-06T13:33:52.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:52.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:33:52.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:33:52.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:52 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:52.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:52 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/3988065694"}]': finished 2024-05-06T13:33:52.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:52 smithi170 ceph-mon[23272]: osdmap e82: 8 total, 8 up, 8 in 2024-05-06T13:33:52.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:52 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:52.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:52 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/1424213713' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/371454062"}]: dispatch 2024-05-06T13:33:52.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:52 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/371454062"}]: dispatch 2024-05-06T13:33:52.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:52 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:52.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:52 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:52.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:52 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:33:52.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:52 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:33:52.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:52 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:52.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:52 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi077"}]: dispatch 2024-05-06T13:33:52.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:52 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:52.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:52 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:33:52.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:52 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:33:52.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:52 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:33:53.454 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:53 smithi077 ceph-mon[27166]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:33:53.454 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:53 smithi077 ceph-mon[27166]: Adding iSCSI gateway http://:@172.21.15.77:5000 to Dashboard 2024-05-06T13:33:53.455 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:53 smithi077 ceph-mon[27166]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:33:53.455 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:53 smithi077 ceph-mon[27166]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi077"}]: dispatch 2024-05-06T13:33:53.455 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:53 smithi077 ceph-mon[27166]: Checking dashboard <-> RGW credentials 2024-05-06T13:33:53.455 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:53 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/371454062"}]': finished 2024-05-06T13:33:53.455 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:53 smithi077 ceph-mon[27166]: osdmap e83: 8 total, 8 up, 8 in 2024-05-06T13:33:53.455 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:53 smithi077 ceph-mon[27166]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 2 op/s; 11 B/s, 0 objects/s recovering 2024-05-06T13:33:53.455 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:53 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3770548783' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/3988065694"}]: dispatch 2024-05-06T13:33:53.455 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:53 smithi077 ceph-mon[22428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:33:53.455 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:53 smithi077 ceph-mon[22428]: Adding iSCSI gateway http://:@172.21.15.77:5000 to Dashboard 2024-05-06T13:33:53.455 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:53 smithi077 ceph-mon[22428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:33:53.455 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:53 smithi077 ceph-mon[22428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi077"}]: dispatch 2024-05-06T13:33:53.456 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:53 smithi077 ceph-mon[22428]: Checking dashboard <-> RGW credentials 2024-05-06T13:33:53.456 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:53 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/371454062"}]': finished 2024-05-06T13:33:53.456 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:53 smithi077 ceph-mon[22428]: osdmap e83: 8 total, 8 up, 8 in 2024-05-06T13:33:53.456 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:53 smithi077 ceph-mon[22428]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 2 op/s; 11 B/s, 0 objects/s recovering 2024-05-06T13:33:53.456 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:53 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3770548783' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/3988065694"}]: dispatch 2024-05-06T13:33:53.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:53 smithi170 ceph-mon[23272]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:33:53.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:53 smithi170 ceph-mon[23272]: Adding iSCSI gateway http://:@172.21.15.77:5000 to Dashboard 2024-05-06T13:33:53.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:53 smithi170 ceph-mon[23272]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:33:53.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:53 smithi170 ceph-mon[23272]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi077"}]: dispatch 2024-05-06T13:33:53.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:53 smithi170 ceph-mon[23272]: Checking dashboard <-> RGW credentials 2024-05-06T13:33:53.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:53 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/371454062"}]': finished 2024-05-06T13:33:53.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:53 smithi170 ceph-mon[23272]: osdmap e83: 8 total, 8 up, 8 in 2024-05-06T13:33:53.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:53 smithi170 ceph-mon[23272]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 2 op/s; 11 B/s, 0 objects/s recovering 2024-05-06T13:33:53.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:53 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/3770548783' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/3988065694"}]: dispatch 2024-05-06T13:33:53.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:33:53.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:33:53.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:33:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:33:53.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs" 2024-05-06T13:33:54.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:33:54 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:33:54] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:33:54.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:54 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/3770548783' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/3988065694"}]': finished 2024-05-06T13:33:54.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:54 smithi170 ceph-mon[23272]: osdmap e84: 8 total, 8 up, 8 in 2024-05-06T13:33:54.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:54 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/1343647977' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/4134859885"}]: dispatch 2024-05-06T13:33:54.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3770548783' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/3988065694"}]': finished 2024-05-06T13:33:54.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:54 smithi077 ceph-mon[27166]: osdmap e84: 8 total, 8 up, 8 in 2024-05-06T13:33:54.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/1343647977' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/4134859885"}]: dispatch 2024-05-06T13:33:54.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:54 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3770548783' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/3988065694"}]': finished 2024-05-06T13:33:54.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:54 smithi077 ceph-mon[22428]: osdmap e84: 8 total, 8 up, 8 in 2024-05-06T13:33:54.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:54 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/1343647977' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/4134859885"}]: dispatch 2024-05-06T13:33:55.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:33:54 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:33:54] "GET /metrics HTTP/1.1" 200 207956 "" "Prometheus/2.33.4" 2024-05-06T13:33:55.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:55 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/1343647977' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/4134859885"}]': finished 2024-05-06T13:33:55.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:55 smithi077 ceph-mon[22428]: osdmap e85: 8 total, 8 up, 8 in 2024-05-06T13:33:55.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:55 smithi077 ceph-mon[22428]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 2 op/s; 11 B/s, 0 objects/s recovering 2024-05-06T13:33:55.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:55 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2222280389' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/1922446457"}]: dispatch 2024-05-06T13:33:55.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:55 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/1343647977' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/4134859885"}]': finished 2024-05-06T13:33:55.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:55 smithi077 ceph-mon[27166]: osdmap e85: 8 total, 8 up, 8 in 2024-05-06T13:33:55.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:55 smithi077 ceph-mon[27166]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 2 op/s; 11 B/s, 0 objects/s recovering 2024-05-06T13:33:55.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:55 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2222280389' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/1922446457"}]: dispatch 2024-05-06T13:33:55.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:55 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/1343647977' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/4134859885"}]': finished 2024-05-06T13:33:55.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:55 smithi170 ceph-mon[23272]: osdmap e85: 8 total, 8 up, 8 in 2024-05-06T13:33:55.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:55 smithi170 ceph-mon[23272]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 2 op/s; 11 B/s, 0 objects/s recovering 2024-05-06T13:33:55.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:55 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2222280389' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/1922446457"}]: dispatch 2024-05-06T13:33:56.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:56 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2222280389' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/1922446457"}]': finished 2024-05-06T13:33:56.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:56 smithi077 ceph-mon[22428]: osdmap e86: 8 total, 8 up, 8 in 2024-05-06T13:33:56.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:56 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2754361283' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/554686488"}]: dispatch 2024-05-06T13:33:56.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:56 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/554686488"}]: dispatch 2024-05-06T13:33:56.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:56 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2222280389' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/1922446457"}]': finished 2024-05-06T13:33:56.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:56 smithi077 ceph-mon[27166]: osdmap e86: 8 total, 8 up, 8 in 2024-05-06T13:33:56.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:56 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2754361283' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/554686488"}]: dispatch 2024-05-06T13:33:56.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:56 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/554686488"}]: dispatch 2024-05-06T13:33:56.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:56 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2222280389' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/1922446457"}]': finished 2024-05-06T13:33:56.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:56 smithi170 ceph-mon[23272]: osdmap e86: 8 total, 8 up, 8 in 2024-05-06T13:33:56.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:56 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2754361283' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/554686488"}]: dispatch 2024-05-06T13:33:56.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:56 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/554686488"}]: dispatch 2024-05-06T13:33:57.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:57 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/554686488"}]': finished 2024-05-06T13:33:57.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:57 smithi077 ceph-mon[22428]: osdmap e87: 8 total, 8 up, 8 in 2024-05-06T13:33:57.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:57 smithi077 ceph-mon[22428]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 66 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-05-06T13:33:57.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:57 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/125072569' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/3193136642"}]: dispatch 2024-05-06T13:33:57.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:57 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/3193136642"}]: dispatch 2024-05-06T13:33:57.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:57 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/554686488"}]': finished 2024-05-06T13:33:57.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:57 smithi077 ceph-mon[27166]: osdmap e87: 8 total, 8 up, 8 in 2024-05-06T13:33:57.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:57 smithi077 ceph-mon[27166]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 66 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-05-06T13:33:57.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:57 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/125072569' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/3193136642"}]: dispatch 2024-05-06T13:33:57.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:57 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/3193136642"}]: dispatch 2024-05-06T13:33:57.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:57 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/554686488"}]': finished 2024-05-06T13:33:57.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:57 smithi170 ceph-mon[23272]: osdmap e87: 8 total, 8 up, 8 in 2024-05-06T13:33:57.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:57 smithi170 ceph-mon[23272]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 66 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-05-06T13:33:57.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:57 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/125072569' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/3193136642"}]: dispatch 2024-05-06T13:33:57.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:57 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/3193136642"}]: dispatch 2024-05-06T13:33:58.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:58 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/3193136642"}]': finished 2024-05-06T13:33:58.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:58 smithi077 ceph-mon[22428]: osdmap e88: 8 total, 8 up, 8 in 2024-05-06T13:33:58.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:58 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/363801396' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2039164702"}]: dispatch 2024-05-06T13:33:58.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:58 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2039164702"}]: dispatch 2024-05-06T13:33:58.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:58 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/3193136642"}]': finished 2024-05-06T13:33:58.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:58 smithi077 ceph-mon[27166]: osdmap e88: 8 total, 8 up, 8 in 2024-05-06T13:33:58.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:58 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/363801396' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2039164702"}]: dispatch 2024-05-06T13:33:58.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:58 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2039164702"}]: dispatch 2024-05-06T13:33:58.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:58 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/3193136642"}]': finished 2024-05-06T13:33:58.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:58 smithi170 ceph-mon[23272]: osdmap e88: 8 total, 8 up, 8 in 2024-05-06T13:33:58.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:58 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/363801396' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2039164702"}]: dispatch 2024-05-06T13:33:58.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:58 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2039164702"}]: dispatch 2024-05-06T13:33:59.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:59 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:33:59.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:59 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2039164702"}]': finished 2024-05-06T13:33:59.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:59 smithi077 ceph-mon[22428]: osdmap e89: 8 total, 8 up, 8 in 2024-05-06T13:33:59.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:59 smithi077 ceph-mon[22428]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 66 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-05-06T13:33:59.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:33:59 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2320639475' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1112825224"}]: dispatch 2024-05-06T13:33:59.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:59 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:33:59.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:59 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2039164702"}]': finished 2024-05-06T13:33:59.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:59 smithi077 ceph-mon[27166]: osdmap e89: 8 total, 8 up, 8 in 2024-05-06T13:33:59.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:59 smithi077 ceph-mon[27166]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 66 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-05-06T13:33:59.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:33:59 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2320639475' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1112825224"}]: dispatch 2024-05-06T13:33:59.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:59 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:33:59.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:59 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2039164702"}]': finished 2024-05-06T13:33:59.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:59 smithi170 ceph-mon[23272]: osdmap e89: 8 total, 8 up, 8 in 2024-05-06T13:33:59.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:59 smithi170 ceph-mon[23272]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 66 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-05-06T13:33:59.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:33:59 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2320639475' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1112825224"}]: dispatch 2024-05-06T13:34:00.521 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:00 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2320639475' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1112825224"}]': finished 2024-05-06T13:34:00.521 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:00 smithi077 ceph-mon[27166]: osdmap e90: 8 total, 8 up, 8 in 2024-05-06T13:34:00.521 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:00 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/1304194825' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/2007005358"}]: dispatch 2024-05-06T13:34:00.521 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:00 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/2007005358"}]: dispatch 2024-05-06T13:34:00.521 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:00 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2320639475' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1112825224"}]': finished 2024-05-06T13:34:00.521 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:00 smithi077 ceph-mon[22428]: osdmap e90: 8 total, 8 up, 8 in 2024-05-06T13:34:00.522 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:00 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/1304194825' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/2007005358"}]: dispatch 2024-05-06T13:34:00.522 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:00 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/2007005358"}]: dispatch 2024-05-06T13:34:00.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:00 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2320639475' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1112825224"}]': finished 2024-05-06T13:34:00.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:00 smithi170 ceph-mon[23272]: osdmap e90: 8 total, 8 up, 8 in 2024-05-06T13:34:00.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:00 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/1304194825' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/2007005358"}]: dispatch 2024-05-06T13:34:00.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:00 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/2007005358"}]: dispatch 2024-05-06T13:34:01.519 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:01 smithi077 ceph-mon[27166]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 66 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:01.519 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:01 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/2007005358"}]': finished 2024-05-06T13:34:01.519 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:01 smithi077 ceph-mon[27166]: osdmap e91: 8 total, 8 up, 8 in 2024-05-06T13:34:01.519 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:01 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/37444828' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/1922446457"}]: dispatch 2024-05-06T13:34:01.519 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:01 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/1922446457"}]: dispatch 2024-05-06T13:34:01.519 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:01 smithi077 ceph-mon[22428]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 66 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:01.520 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:01 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/2007005358"}]': finished 2024-05-06T13:34:01.520 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:01 smithi077 ceph-mon[22428]: osdmap e91: 8 total, 8 up, 8 in 2024-05-06T13:34:01.520 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:01 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/37444828' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/1922446457"}]: dispatch 2024-05-06T13:34:01.520 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:01 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/1922446457"}]: dispatch 2024-05-06T13:34:01.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:01 smithi170 ceph-mon[23272]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 66 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:01.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:01 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/2007005358"}]': finished 2024-05-06T13:34:01.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:01 smithi170 ceph-mon[23272]: osdmap e91: 8 total, 8 up, 8 in 2024-05-06T13:34:01.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:01 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/37444828' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/1922446457"}]: dispatch 2024-05-06T13:34:01.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:01 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/1922446457"}]: dispatch 2024-05-06T13:34:02.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:02 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/1922446457"}]': finished 2024-05-06T13:34:02.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:02 smithi077 ceph-mon[27166]: osdmap e92: 8 total, 8 up, 8 in 2024-05-06T13:34:02.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:02 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2453120701' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/3698539339"}]: dispatch 2024-05-06T13:34:02.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:02 smithi077 ceph-mon[27166]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:02.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:02 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/1922446457"}]': finished 2024-05-06T13:34:02.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:02 smithi077 ceph-mon[22428]: osdmap e92: 8 total, 8 up, 8 in 2024-05-06T13:34:02.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:02 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2453120701' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/3698539339"}]: dispatch 2024-05-06T13:34:02.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:02 smithi077 ceph-mon[22428]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:02.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:02 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/1922446457"}]': finished 2024-05-06T13:34:02.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:02 smithi170 ceph-mon[23272]: osdmap e92: 8 total, 8 up, 8 in 2024-05-06T13:34:02.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:02 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2453120701' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/3698539339"}]: dispatch 2024-05-06T13:34:02.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:02 smithi170 ceph-mon[23272]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:03.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:03 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2453120701' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/3698539339"}]': finished 2024-05-06T13:34:03.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:03 smithi077 ceph-mon[27166]: osdmap e93: 8 total, 8 up, 8 in 2024-05-06T13:34:03.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:03 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3244217488' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/2007005358"}]: dispatch 2024-05-06T13:34:03.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:03 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/2007005358"}]: dispatch 2024-05-06T13:34:03.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:03 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2453120701' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/3698539339"}]': finished 2024-05-06T13:34:03.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:03 smithi077 ceph-mon[22428]: osdmap e93: 8 total, 8 up, 8 in 2024-05-06T13:34:03.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:03 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3244217488' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/2007005358"}]: dispatch 2024-05-06T13:34:03.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:03 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/2007005358"}]: dispatch 2024-05-06T13:34:03.803 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:34:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=error ts=2024-05-06T13:34:03.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:34:03.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:03 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2453120701' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/3698539339"}]': finished 2024-05-06T13:34:03.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:03 smithi170 ceph-mon[23272]: osdmap e93: 8 total, 8 up, 8 in 2024-05-06T13:34:03.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:03 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/3244217488' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/2007005358"}]: dispatch 2024-05-06T13:34:03.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:03 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/2007005358"}]: dispatch 2024-05-06T13:34:04.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:34:04 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:34:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:34:04.802 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:34:04 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:34:04] "GET /metrics HTTP/1.1" 200 208107 "" "Prometheus/2.33.4" 2024-05-06T13:34:04.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:04 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/2007005358"}]': finished 2024-05-06T13:34:04.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:04 smithi077 ceph-mon[27166]: osdmap e94: 8 total, 8 up, 8 in 2024-05-06T13:34:04.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:04 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/818269709' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/461831909"}]: dispatch 2024-05-06T13:34:04.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:04 smithi077 ceph-mon[27166]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:34:04.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:04 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/2007005358"}]': finished 2024-05-06T13:34:04.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:04 smithi077 ceph-mon[22428]: osdmap e94: 8 total, 8 up, 8 in 2024-05-06T13:34:04.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:04 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/818269709' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/461831909"}]: dispatch 2024-05-06T13:34:04.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:04 smithi077 ceph-mon[22428]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:34:04.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:04 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/2007005358"}]': finished 2024-05-06T13:34:04.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:04 smithi170 ceph-mon[23272]: osdmap e94: 8 total, 8 up, 8 in 2024-05-06T13:34:04.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:04 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/818269709' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/461831909"}]: dispatch 2024-05-06T13:34:04.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:04 smithi170 ceph-mon[23272]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:34:05.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:05 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/818269709' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/461831909"}]': finished 2024-05-06T13:34:05.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:05 smithi077 ceph-mon[22428]: osdmap e95: 8 total, 8 up, 8 in 2024-05-06T13:34:05.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:05 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/818269709' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/461831909"}]': finished 2024-05-06T13:34:05.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:05 smithi077 ceph-mon[27166]: osdmap e95: 8 total, 8 up, 8 in 2024-05-06T13:34:05.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:05 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/818269709' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/461831909"}]': finished 2024-05-06T13:34:05.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:05 smithi170 ceph-mon[23272]: osdmap e95: 8 total, 8 up, 8 in 2024-05-06T13:34:06.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:06 smithi077 ceph-mon[22428]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T13:34:06.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:06 smithi077 ceph-mon[27166]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T13:34:06.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:06 smithi170 ceph-mon[23272]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T13:34:08.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:08 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:34:08.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:08 smithi077 ceph-mon[22428]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:08.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:08 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:34:08.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:08 smithi077 ceph-mon[27166]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:08.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:08 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:34:08.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:08 smithi170 ceph-mon[23272]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:10.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:10 smithi077 ceph-mon[22428]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-06T13:34:10.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:10 smithi077 ceph-mon[27166]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-06T13:34:10.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:10 smithi170 ceph-mon[23272]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-06T13:34:12.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:12 smithi077 ceph-mon[22428]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T13:34:12.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:12 smithi077 ceph-mon[27166]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T13:34:12.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:12 smithi170 ceph-mon[23272]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T13:34:14.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:34:14 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:34:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:34:14.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:14 smithi077 ceph-mon[22428]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T13:34:14.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:14 smithi077 ceph-mon[27166]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T13:34:14.802 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:34:14 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:34:14] "GET /metrics HTTP/1.1" 200 208078 "" "Prometheus/2.33.4" 2024-05-06T13:34:14.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:14 smithi170 ceph-mon[23272]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T13:34:16.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:16 smithi077 ceph-mon[22428]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-06T13:34:16.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:16 smithi077 ceph-mon[27166]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-06T13:34:16.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:16 smithi170 ceph-mon[23272]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-06T13:34:18.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:18 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:34:18.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:18 smithi077 ceph-mon[22428]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:18.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:18 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:34:18.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:18 smithi077 ceph-mon[27166]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:18.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:18 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:34:18.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:18 smithi170 ceph-mon[23272]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:20.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:20 smithi077 ceph-mon[22428]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:20.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:20 smithi077 ceph-mon[27166]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:20.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:20 smithi170 ceph-mon[23272]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:22.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:22 smithi077 ceph-mon[22428]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:22.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:22 smithi077 ceph-mon[27166]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:22.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:22 smithi170 ceph-mon[23272]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:24.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:34:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:34:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:34:24.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:24 smithi077 ceph-mon[22428]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:24.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:24 smithi077 ceph-mon[27166]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:24.802 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:34:24 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:34:24] "GET /metrics HTTP/1.1" 200 208078 "" "Prometheus/2.33.4" 2024-05-06T13:34:24.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:24 smithi170 ceph-mon[23272]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:26.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:26 smithi077 ceph-mon[22428]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:26.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:26 smithi077 ceph-mon[27166]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:26.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:26 smithi170 ceph-mon[23272]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:28.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:28 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:34:28.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:28 smithi077 ceph-mon[22428]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:28.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:28 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:34:28.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:28 smithi077 ceph-mon[27166]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:28.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:28 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:34:28.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:28 smithi170 ceph-mon[23272]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:30.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:30 smithi077 ceph-mon[22428]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:30.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:30 smithi077 ceph-mon[27166]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:30.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:30 smithi170 ceph-mon[23272]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:32.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:32 smithi077 ceph-mon[22428]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:32.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:32 smithi077 ceph-mon[27166]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:32.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:32 smithi170 ceph-mon[23272]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:34.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:34:34 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:34:34] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:34:34.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:34 smithi077 ceph-mon[22428]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:34.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:34 smithi077 ceph-mon[27166]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:34.802 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:34:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:34:34] "GET /metrics HTTP/1.1" 200 208047 "" "Prometheus/2.33.4" 2024-05-06T13:34:34.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:34 smithi170 ceph-mon[23272]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:36.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:36 smithi077 ceph-mon[22428]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:36.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:36 smithi077 ceph-mon[27166]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:36.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:36 smithi170 ceph-mon[23272]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:38.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:38 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:34:38.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:38 smithi077 ceph-mon[27166]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:38.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:38 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:34:38.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:38 smithi077 ceph-mon[22428]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:38.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:38 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:34:38.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:38 smithi170 ceph-mon[23272]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:40.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:40 smithi077 ceph-mon[22428]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:40.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:40 smithi077 ceph-mon[27166]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:40.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:40 smithi170 ceph-mon[23272]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:42.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:42 smithi077 ceph-mon[22428]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:42.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:42 smithi077 ceph-mon[27166]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:42.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:42 smithi170 ceph-mon[23272]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:44.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:34:44 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:34:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:34:44.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:44 smithi077 ceph-mon[22428]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:44.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:34:44.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:34:44.802 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:34:44 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:34:44] "GET /metrics HTTP/1.1" 200 208005 "" "Prometheus/2.33.4" 2024-05-06T13:34:44.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:44 smithi077 ceph-mon[27166]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:44.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:34:44.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:34:44.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:44 smithi170 ceph-mon[23272]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:44.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:34:44.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:34:46.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:46 smithi077 ceph-mon[22428]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:46.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:46 smithi077 ceph-mon[27166]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:46.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:46 smithi170 ceph-mon[23272]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:48.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:48 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:34:48.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:48 smithi077 ceph-mon[27166]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:48.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:48 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:34:48.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:48 smithi077 ceph-mon[22428]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:48.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:48 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:34:48.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:48 smithi170 ceph-mon[23272]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:50.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:50 smithi077 ceph-mon[22428]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:50.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:50 smithi077 ceph-mon[27166]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:50.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:50 smithi170 ceph-mon[23272]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:52.000 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:51 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:34:52.000 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:51 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:34:52.000 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:51 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:34:52.000 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:51 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:34:52.004 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:51 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:34:52.004 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:51 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:34:53.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:53 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:34:53.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:53 smithi077 ceph-mon[27166]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:53.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:53 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:34:53.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:53 smithi077 ceph-mon[22428]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:53.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:53 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:34:53.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:53 smithi170 ceph-mon[23272]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:54.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:34:54 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:34:54] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:34:54.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:54 smithi077 ceph-mon[22428]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:54.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:54 smithi077 ceph-mon[27166]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:54.806 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:34:54 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:34:54] "GET /metrics HTTP/1.1" 200 208005 "" "Prometheus/2.33.4" 2024-05-06T13:34:54.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:54 smithi170 ceph-mon[23272]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:56.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:56 smithi077 ceph-mon[22428]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:56.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:56 smithi077 ceph-mon[27166]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:56.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:56 smithi170 ceph-mon[23272]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:34:58.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:58 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:34:58.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:34:58 smithi077 ceph-mon[22428]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:58.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:58 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:34:58.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:34:58 smithi077 ceph-mon[27166]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:34:58.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:58 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:34:58.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:34:58 smithi170 ceph-mon[23272]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:00.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:00 smithi077 ceph-mon[22428]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:00.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:00 smithi077 ceph-mon[27166]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:00.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:00 smithi170 ceph-mon[23272]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:02.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:02 smithi077 ceph-mon[22428]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:02.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:02 smithi077 ceph-mon[27166]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:02.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:02 smithi170 ceph-mon[23272]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:04.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:35:04 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:35:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:35:04.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:04 smithi077 ceph-mon[22428]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:04.802 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:35:04 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:35:04] "GET /metrics HTTP/1.1" 200 208018 "" "Prometheus/2.33.4" 2024-05-06T13:35:04.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:04 smithi077 ceph-mon[27166]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:04.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:04 smithi170 ceph-mon[23272]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:06.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:06 smithi077 ceph-mon[22428]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:06.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:06 smithi077 ceph-mon[27166]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:06.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:06 smithi170 ceph-mon[23272]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:08.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:08 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:35:08.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:08 smithi077 ceph-mon[22428]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:08.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:08 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:35:08.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:08 smithi077 ceph-mon[27166]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:08.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:08 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:35:08.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:08 smithi170 ceph-mon[23272]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:10.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:10 smithi077 ceph-mon[22428]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:10.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:10 smithi077 ceph-mon[27166]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:10.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:10 smithi170 ceph-mon[23272]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:12.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:12 smithi077 ceph-mon[22428]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:12.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:12 smithi077 ceph-mon[27166]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:12.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:12 smithi170 ceph-mon[23272]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:14.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:35:14 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:35:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:35:14.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:14 smithi077 ceph-mon[22428]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:14.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:14 smithi077 ceph-mon[27166]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:14.803 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:35:14 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:35:14] "GET /metrics HTTP/1.1" 200 208011 "" "Prometheus/2.33.4" 2024-05-06T13:35:14.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:14 smithi170 ceph-mon[23272]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:16.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:16 smithi077 ceph-mon[22428]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:16.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:16 smithi077 ceph-mon[27166]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:16.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:16 smithi170 ceph-mon[23272]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:18.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:18 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:35:18.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:18 smithi077 ceph-mon[22428]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:18.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:18 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:35:18.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:18 smithi077 ceph-mon[27166]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:18.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:18 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:35:18.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:18 smithi170 ceph-mon[23272]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:20.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:20 smithi077 ceph-mon[22428]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:20.806 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:20 smithi077 ceph-mon[27166]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:20.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:20 smithi170 ceph-mon[23272]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:22.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:22 smithi077 ceph-mon[22428]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:22.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:22 smithi077 ceph-mon[27166]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:22.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:22 smithi170 ceph-mon[23272]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:24.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:35:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:35:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:35:24.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:24 smithi077 ceph-mon[22428]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:24.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:24 smithi077 ceph-mon[27166]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:24.803 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:35:24 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:35:24] "GET /metrics HTTP/1.1" 200 208011 "" "Prometheus/2.33.4" 2024-05-06T13:35:24.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:24 smithi170 ceph-mon[23272]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:26.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:26 smithi077 ceph-mon[27166]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:26.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:26 smithi077 ceph-mon[22428]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:26.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:26 smithi170 ceph-mon[23272]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:28.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:28 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:35:28.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:28 smithi077 ceph-mon[22428]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:28.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:28 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:35:28.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:28 smithi077 ceph-mon[27166]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:28.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:28 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:35:28.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:28 smithi170 ceph-mon[23272]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:30.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:30 smithi077 ceph-mon[22428]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:30.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:30 smithi077 ceph-mon[27166]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:30.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:30 smithi170 ceph-mon[23272]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:32.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:32 smithi077 ceph-mon[22428]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:32.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:32 smithi077 ceph-mon[27166]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:32.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:32 smithi170 ceph-mon[23272]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:34.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:35:34 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:35:34] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:35:34.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:34 smithi077 ceph-mon[22428]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:34.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:34 smithi077 ceph-mon[27166]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:34.803 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:35:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:35:34] "GET /metrics HTTP/1.1" 200 208019 "" "Prometheus/2.33.4" 2024-05-06T13:35:34.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:34 smithi170 ceph-mon[23272]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:36.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:36 smithi077 ceph-mon[22428]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:36.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:36 smithi077 ceph-mon[27166]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:36.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:36 smithi170 ceph-mon[23272]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:38.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:38 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:35:38.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:38 smithi077 ceph-mon[27166]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:38.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:38 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:35:38.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:38 smithi077 ceph-mon[22428]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:38.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:38 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:35:38.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:38 smithi170 ceph-mon[23272]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:40.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:40 smithi077 ceph-mon[22428]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:40.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:40 smithi077 ceph-mon[27166]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:40.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:40 smithi170 ceph-mon[23272]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:42.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:42 smithi077 ceph-mon[22428]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:42.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:42 smithi077 ceph-mon[27166]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:42.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:42 smithi170 ceph-mon[23272]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:44.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:35:44 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:35:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:35:44.802 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:35:44 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:35:44] "GET /metrics HTTP/1.1" 200 208015 "" "Prometheus/2.33.4" 2024-05-06T13:35:44.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:44 smithi077 ceph-mon[22428]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:44.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:35:44.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:35:44.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:44 smithi077 ceph-mon[27166]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:44.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:35:44.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:35:44.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:44 smithi170 ceph-mon[23272]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:44.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:35:44.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:35:46.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:46 smithi077 ceph-mon[27166]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:46.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:46 smithi077 ceph-mon[22428]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:46.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:46 smithi170 ceph-mon[23272]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:48.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:48 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:35:48.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:48 smithi077 ceph-mon[27166]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:48.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:48 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:35:48.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:48 smithi077 ceph-mon[22428]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:48.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:48 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:35:48.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:48 smithi170 ceph-mon[23272]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:50.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:50 smithi077 ceph-mon[22428]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:50.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:50 smithi077 ceph-mon[27166]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:50.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:50 smithi170 ceph-mon[23272]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:52.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:52 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:35:52.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:52 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:35:52.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:52 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:35:52.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:52 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:35:52.321 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:52 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:35:52.321 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:52 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:35:53.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:53 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:35:53.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:53 smithi077 ceph-mon[22428]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:53.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:53 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:35:53.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:53 smithi077 ceph-mon[27166]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:53.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:53 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:35:53.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:53 smithi170 ceph-mon[23272]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:54.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:35:54 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:35:54] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:35:54.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:54 smithi077 ceph-mon[22428]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:54.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:54 smithi077 ceph-mon[27166]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:54.802 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:35:54 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:35:54] "GET /metrics HTTP/1.1" 200 208015 "" "Prometheus/2.33.4" 2024-05-06T13:35:54.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:54 smithi170 ceph-mon[23272]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:56.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:56 smithi077 ceph-mon[22428]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:56.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:56 smithi077 ceph-mon[27166]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:56.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:56 smithi170 ceph-mon[23272]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:35:58.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:58 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:35:58.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:35:58 smithi077 ceph-mon[27166]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:58.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:58 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:35:58.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:35:58 smithi077 ceph-mon[22428]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:35:58.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:58 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:35:58.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:35:58 smithi170 ceph-mon[23272]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:00.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:00 smithi077 ceph-mon[22428]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:00.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:00 smithi077 ceph-mon[27166]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:00.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:00 smithi170 ceph-mon[23272]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:02.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:02 smithi077 ceph-mon[22428]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:02.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:02 smithi077 ceph-mon[27166]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:02.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:02 smithi170 ceph-mon[23272]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:04.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:36:04 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:36:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:36:04.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:04 smithi077 ceph-mon[22428]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:04.802 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:36:04 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:36:04] "GET /metrics HTTP/1.1" 200 208015 "" "Prometheus/2.33.4" 2024-05-06T13:36:04.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:04 smithi077 ceph-mon[27166]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:04.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:04 smithi170 ceph-mon[23272]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:06.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:06 smithi077 ceph-mon[22428]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:06.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:06 smithi077 ceph-mon[27166]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:06.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:06 smithi170 ceph-mon[23272]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:08.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:08 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:36:08.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:08 smithi077 ceph-mon[22428]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:08.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:08 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:36:08.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:08 smithi077 ceph-mon[27166]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:08.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:08 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:36:08.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:08 smithi170 ceph-mon[23272]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:10.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:10 smithi077 ceph-mon[22428]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:10.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:10 smithi077 ceph-mon[27166]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:10.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:10 smithi170 ceph-mon[23272]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:12.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:12 smithi077 ceph-mon[22428]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:12.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:12 smithi077 ceph-mon[27166]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:12.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:12 smithi170 ceph-mon[23272]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:14.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:36:14 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:36:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:36:14.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:14 smithi077 ceph-mon[22428]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:14.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:14 smithi077 ceph-mon[27166]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:14.803 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:36:14 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:36:14] "GET /metrics HTTP/1.1" 200 207974 "" "Prometheus/2.33.4" 2024-05-06T13:36:14.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:14 smithi170 ceph-mon[23272]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:16.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:16 smithi077 ceph-mon[22428]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:16.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:16 smithi077 ceph-mon[27166]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:16.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:16 smithi170 ceph-mon[23272]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:18.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:18 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:36:18.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:18 smithi077 ceph-mon[27166]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:18.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:18 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:36:18.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:18 smithi077 ceph-mon[22428]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:18.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:18 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:36:18.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:18 smithi170 ceph-mon[23272]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:20.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:20 smithi077 ceph-mon[22428]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:20.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:20 smithi077 ceph-mon[27166]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:20.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:20 smithi170 ceph-mon[23272]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:22.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:22 smithi077 ceph-mon[22428]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:22.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:22 smithi077 ceph-mon[27166]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:22.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:22 smithi170 ceph-mon[23272]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:24.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:36:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:36:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:36:24.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:24 smithi077 ceph-mon[22428]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:24.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:24 smithi077 ceph-mon[27166]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:24.803 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:36:24 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:36:24] "GET /metrics HTTP/1.1" 200 207974 "" "Prometheus/2.33.4" 2024-05-06T13:36:24.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:24 smithi170 ceph-mon[23272]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:26.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:26 smithi170 ceph-mon[23272]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:27.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:26 smithi077 ceph-mon[22428]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:27.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:26 smithi077 ceph-mon[27166]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:28.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:28 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:36:28.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:28 smithi077 ceph-mon[22428]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:28.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:28 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:36:28.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:28 smithi077 ceph-mon[27166]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:28.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:28 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:36:28.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:28 smithi170 ceph-mon[23272]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:30.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:30 smithi077 ceph-mon[22428]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:30.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:30 smithi077 ceph-mon[27166]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:30.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:30 smithi170 ceph-mon[23272]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:32.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:32 smithi170 ceph-mon[23272]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:33.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:32 smithi077 ceph-mon[22428]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:33.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:32 smithi077 ceph-mon[27166]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:34.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:36:34 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:36:34] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:36:34.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:34 smithi170 ceph-mon[23272]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:35.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:34 smithi077 ceph-mon[22428]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:35.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:34 smithi077 ceph-mon[27166]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:35.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:36:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:36:34] "GET /metrics HTTP/1.1" 200 207964 "" "Prometheus/2.33.4" 2024-05-06T13:36:36.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:36 smithi170 ceph-mon[23272]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:37.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:36 smithi077 ceph-mon[22428]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:37.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:36 smithi077 ceph-mon[27166]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:38.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:38 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:36:38.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:38 smithi170 ceph-mon[23272]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:39.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:38 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:36:39.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:38 smithi077 ceph-mon[22428]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:39.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:38 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:36:39.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:38 smithi077 ceph-mon[27166]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:40.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:40 smithi077 ceph-mon[22428]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:40.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:40 smithi077 ceph-mon[27166]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:40.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:40 smithi170 ceph-mon[23272]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:42.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:42 smithi170 ceph-mon[23272]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:43.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:42 smithi077 ceph-mon[22428]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:43.083 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:42 smithi077 ceph-mon[27166]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:44.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:36:44 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:36:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:36:44.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:44 smithi170 ceph-mon[23272]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:44.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:36:44.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:36:45.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:36:44 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:36:44] "GET /metrics HTTP/1.1" 200 207970 "" "Prometheus/2.33.4" 2024-05-06T13:36:45.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:44 smithi077 ceph-mon[22428]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:45.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:36:45.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:36:45.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:44 smithi077 ceph-mon[27166]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:45.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:36:45.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:36:46.194 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-05-06T13:36:46.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:46 smithi077 ceph-mon[22428]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:46.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:46 smithi077 ceph-mon[27166]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:46.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:46 smithi170 ceph-mon[23272]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:48.656 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:48 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:36:48.656 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:48 smithi077 ceph-mon[22428]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:48.656 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:48 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:36:48.656 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:48 smithi077 ceph-mon[27166]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:48.685 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-05-06T13:36:48.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:48 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:36:48.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:48 smithi170 ceph-mon[23272]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:50.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:50 smithi077 ceph-mon[22428]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:50.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:50 smithi077 ceph-mon[27166]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:50.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:50 smithi170 ceph-mon[23272]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:51.370 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph config set global log_to_journald false --force' 2024-05-06T13:36:52.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:52 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:36:52.634 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:52 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:36:52.704 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:52 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:36:52.704 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:52 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:36:52.704 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:52 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:36:52.704 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:52 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:36:53.686 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:53 smithi077 ceph-mon[22428]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:53.687 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:53 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:36:53.687 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:53 smithi077 ceph-mon[27166]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:53.687 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:53 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:36:53.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:53 smithi170 ceph-mon[23272]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:53.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:53 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:36:54.077 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch ps' 2024-05-06T13:36:54.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:36:54 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:36:54] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:36:54.662 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:54 smithi077 ceph-mon[27166]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:54.662 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:54 smithi077 ceph-mon[22428]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:54.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:54 smithi170 ceph-mon[23272]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:55.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:36:54 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:36:54] "GET /metrics HTTP/1.1" 200 207970 "" "Prometheus/2.33.4" 2024-05-06T13:36:56.186 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T13:36:56.186 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (3m) 3m ago 4m 15.5M - ba2b418f427c d17996419263 2024-05-06T13:36:56.186 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (3m) 3m ago 3m 48.8M - 8.3.5 dad864ee21e9 4b872c67236d 2024-05-06T13:36:56.186 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (3m) 3m ago 3m 65.9M - 3.5 e1d6a67b021e d705e2e893a0 2024-05-06T13:36:56.186 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443 running (8m) 3m ago 8m 410M - 17.2.0 e1d6a67b021e 82c841abfafd 2024-05-06T13:36:56.187 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:9283 running (10m) 3m ago 10m 453M - 17.2.0 e1d6a67b021e 79a55d163d05 2024-05-06T13:36:56.187 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (10m) 3m ago 10m 45.5M 2048M 17.2.0 e1d6a67b021e ac46f91694f8 2024-05-06T13:36:56.187 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (8m) 3m ago 8m 38.3M 2048M 17.2.0 e1d6a67b021e 24be6fce9d28 2024-05-06T13:36:56.187 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (9m) 3m ago 9m 38.4M 2048M 17.2.0 e1d6a67b021e 83af336ef762 2024-05-06T13:36:56.187 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (4m) 3m ago 4m 15.9M - 1dbe0e931976 eb6f92239232 2024-05-06T13:36:56.187 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (4m) 3m ago 4m 16.5M - 1dbe0e931976 001e61e5f922 2024-05-06T13:36:56.187 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (8m) 3m ago 8m 47.1M 4016M 17.2.0 e1d6a67b021e 2ccee2bb6e96 2024-05-06T13:36:56.187 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (7m) 3m ago 7m 51.2M 4016M 17.2.0 e1d6a67b021e 1d9e6609b585 2024-05-06T13:36:56.187 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (7m) 3m ago 7m 46.3M 4016M 17.2.0 e1d6a67b021e 37a61d295111 2024-05-06T13:36:56.187 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (7m) 3m ago 7m 45.0M 4016M 17.2.0 e1d6a67b021e 8feb7c077aaa 2024-05-06T13:36:56.187 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (6m) 3m ago 6m 47.9M 4272M 17.2.0 e1d6a67b021e f865acc44a06 2024-05-06T13:36:56.187 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (6m) 3m ago 6m 44.1M 4272M 17.2.0 e1d6a67b021e a55bd4122f6f 2024-05-06T13:36:56.187 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (5m) 3m ago 5m 42.9M 4272M 17.2.0 e1d6a67b021e 362938a0b357 2024-05-06T13:36:56.187 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (5m) 3m ago 5m 45.2M 4272M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T13:36:56.188 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (3m) 3m ago 4m 45.6M - 514e6a882f6e 488818ee0077 2024-05-06T13:36:56.188 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (3m) 3m ago 3m 70.3M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T13:36:56.188 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (3m) 3m ago 3m 69.2M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T13:36:56.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:56 smithi170 ceph-mon[23272]: from='client.24880 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:36:56.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:56 smithi170 ceph-mon[23272]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:56.859 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:56 smithi077 ceph-mon[22428]: from='client.24880 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:36:56.859 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:56 smithi077 ceph-mon[22428]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:56.860 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:56 smithi077 ceph-mon[27166]: from='client.24880 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:36:56.860 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:56 smithi077 ceph-mon[27166]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:36:56.883 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph versions' 2024-05-06T13:36:58.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:58 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:36:58.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:58 smithi077 ceph-mon[22428]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:58.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:58 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:36:58.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:58 smithi077 ceph-mon[27166]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:58.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:58 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:36:58.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:58 smithi170 ceph-mon[23272]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:36:58.885 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:36:58.885 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-05-06T13:36:58.885 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-05-06T13:36:58.885 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:36:58.885 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-05-06T13:36:58.886 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T13:36:58.886 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:36:58.886 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-05-06T13:36:58.886 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-06T13:36:58.886 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:36:58.886 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": {}, 2024-05-06T13:36:58.886 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw": { 2024-05-06T13:36:58.886 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T13:36:58.886 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:36:58.886 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-05-06T13:36:58.886 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15 2024-05-06T13:36:58.886 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T13:36:58.886 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:36:59.540 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph -s' 2024-05-06T13:36:59.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:36:59 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2794043737' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:36:59.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:36:59 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2794043737' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:36:59.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:36:59 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2794043737' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:37:00.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:00 smithi170 ceph-mon[23272]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:00.852 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:00 smithi077 ceph-mon[22428]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:00.852 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:00 smithi077 ceph-mon[27166]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:01.641 INFO:teuthology.orchestra.run.smithi077.stdout: cluster: 2024-05-06T13:37:01.641 INFO:teuthology.orchestra.run.smithi077.stdout: id: 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:37:01.641 INFO:teuthology.orchestra.run.smithi077.stdout: health: HEALTH_OK 2024-05-06T13:37:01.641 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:37:01.641 INFO:teuthology.orchestra.run.smithi077.stdout: services: 2024-05-06T13:37:01.641 INFO:teuthology.orchestra.run.smithi077.stdout: mon: 3 daemons, quorum a,c,b (age 8m) 2024-05-06T13:37:01.642 INFO:teuthology.orchestra.run.smithi077.stdout: mgr: y(active, since 5m), standbys: x 2024-05-06T13:37:01.642 INFO:teuthology.orchestra.run.smithi077.stdout: osd: 8 osds: 8 up (since 5m), 8 in (since 5m) 2024-05-06T13:37:01.642 INFO:teuthology.orchestra.run.smithi077.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-05-06T13:37:01.642 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:37:01.642 INFO:teuthology.orchestra.run.smithi077.stdout: data: 2024-05-06T13:37:01.642 INFO:teuthology.orchestra.run.smithi077.stdout: pools: 6 pools, 161 pgs 2024-05-06T13:37:01.642 INFO:teuthology.orchestra.run.smithi077.stdout: objects: 209 objects, 585 KiB 2024-05-06T13:37:01.642 INFO:teuthology.orchestra.run.smithi077.stdout: usage: 68 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:37:01.642 INFO:teuthology.orchestra.run.smithi077.stdout: pgs: 161 active+clean 2024-05-06T13:37:01.642 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:37:01.642 INFO:teuthology.orchestra.run.smithi077.stdout: io: 2024-05-06T13:37:01.642 INFO:teuthology.orchestra.run.smithi077.stdout: client: 1.2 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-05-06T13:37:01.642 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:37:01.937 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:01 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/139553020' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-06T13:37:01.937 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:01 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/139553020' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-06T13:37:02.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:01 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/139553020' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-06T13:37:02.362 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch ls' 2024-05-06T13:37:02.991 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:02 smithi077 ceph-mon[22428]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:02.991 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:02 smithi077 ceph-mon[27166]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:03.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:02 smithi170 ceph-mon[23272]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:04.301 INFO:teuthology.orchestra.run.smithi077.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-05-06T13:37:04.301 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager ?:9093,9094 1/1 3m ago 5m smithi077=a;count:1 2024-05-06T13:37:04.301 INFO:teuthology.orchestra.run.smithi077.stdout:grafana ?:3000 1/1 3m ago 5m smithi170=a;count:1 2024-05-06T13:37:04.301 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo 1/1 3m ago 3m count:1 2024-05-06T13:37:04.301 INFO:teuthology.orchestra.run.smithi077.stdout:mgr 2/2 3m ago 8m smithi077=y;smithi170=x;count:2 2024-05-06T13:37:04.301 INFO:teuthology.orchestra.run.smithi077.stdout:mon 3/3 3m ago 9m smithi077:172.21.15.77=a;smithi077:[v2:172.21.15.77:3301,v1:172.21.15.77:6790]=c;smithi170:172.21.15.170=b;count:3 2024-05-06T13:37:04.301 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter ?:9100 2/2 3m ago 5m smithi077=a;smithi170=b;count:2 2024-05-06T13:37:04.301 INFO:teuthology.orchestra.run.smithi077.stdout:osd 8 3m ago - 2024-05-06T13:37:04.302 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus ?:9095 1/1 3m ago 5m smithi170=a;count:1 2024-05-06T13:37:04.302 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo ?:8000 2/2 3m ago 3m count:2 2024-05-06T13:37:04.325 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:04 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:37:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:37:04.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:04 smithi170 ceph-mon[23272]: from='client.14994 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:37:04.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:04 smithi170 ceph-mon[23272]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:04.944 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:04 smithi077 ceph-mon[22428]: from='client.14994 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:37:04.944 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:04 smithi077 ceph-mon[22428]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:04.944 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:37:04 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:37:04] "GET /metrics HTTP/1.1" 200 207961 "" "Prometheus/2.33.4" 2024-05-06T13:37:04.944 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:04 smithi077 ceph-mon[27166]: from='client.14994 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:37:04.944 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:04 smithi077 ceph-mon[27166]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:04.968 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-05-06T13:37:06.653 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:06 smithi077 ceph-mon[22428]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:06.653 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:06 smithi077 ceph-mon[27166]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:06.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:06 smithi170 ceph-mon[23272]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:07.400 INFO:teuthology.orchestra.run.smithi077.stdout:Scheduled to redeploy mgr.x on host 'smithi170' 2024-05-06T13:37:07.708 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:07 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/1496839953' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-06T13:37:07.708 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:07 smithi077 ceph-mon[22428]: from='client.24910 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:37:07.708 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:07 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:37:07.708 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:07 smithi077 ceph-mon[22428]: Schedule redeploy daemon mgr.x 2024-05-06T13:37:07.708 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:07 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:37:07.708 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:07 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:37:07.709 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:07 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:37:07.709 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:07 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T13:37:07.709 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:07 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T13:37:07.709 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:07 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:37:07.709 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:07 smithi077 ceph-mon[22428]: Deploying daemon mgr.x on smithi170 2024-05-06T13:37:07.709 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:07 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/1496839953' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-06T13:37:07.709 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:07 smithi077 ceph-mon[27166]: from='client.24910 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:37:07.709 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:07 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:37:07.709 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:07 smithi077 ceph-mon[27166]: Schedule redeploy daemon mgr.x 2024-05-06T13:37:07.709 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:07 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:37:07.710 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:07 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:37:07.710 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:07 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:37:07.710 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:07 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T13:37:07.710 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:07 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T13:37:07.710 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:07 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:37:07.710 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:07 smithi077 ceph-mon[27166]: Deploying daemon mgr.x on smithi170 2024-05-06T13:37:07.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:07 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/1496839953' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-06T13:37:07.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:07 smithi170 ceph-mon[23272]: from='client.24910 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:37:07.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:07 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:37:07.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:07 smithi170 ceph-mon[23272]: Schedule redeploy daemon mgr.x 2024-05-06T13:37:07.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:07 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:37:07.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:07 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:37:07.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:07 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:37:07.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:07 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T13:37:07.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:07 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T13:37:07.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:07 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:37:07.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:07 smithi170 ceph-mon[23272]: Deploying daemon mgr.x on smithi170 2024-05-06T13:37:08.092 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch ps --refresh' 2024-05-06T13:37:08.670 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:08 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:37:08.670 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:08 smithi077 ceph-mon[22428]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:08.671 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:08 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:37:08.671 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:08 smithi077 ceph-mon[27166]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:08.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:08 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:37:08.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:08 smithi170 ceph-mon[23272]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:10.117 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T13:37:10.118 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (3m) 3m ago 4m 15.5M - ba2b418f427c d17996419263 2024-05-06T13:37:10.118 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (4m) 3m ago 4m 48.8M - 8.3.5 dad864ee21e9 4b872c67236d 2024-05-06T13:37:10.118 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (3m) 3m ago 3m 65.9M - 3.5 e1d6a67b021e d705e2e893a0 2024-05-06T13:37:10.118 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443 running (8m) 3m ago 8m 410M - 17.2.0 e1d6a67b021e 82c841abfafd 2024-05-06T13:37:10.118 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:9283 running (10m) 3m ago 10m 453M - 17.2.0 e1d6a67b021e 79a55d163d05 2024-05-06T13:37:10.118 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (10m) 3m ago 10m 45.5M 2048M 17.2.0 e1d6a67b021e ac46f91694f8 2024-05-06T13:37:10.118 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (9m) 3m ago 9m 38.3M 2048M 17.2.0 e1d6a67b021e 24be6fce9d28 2024-05-06T13:37:10.118 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (9m) 3m ago 9m 38.4M 2048M 17.2.0 e1d6a67b021e 83af336ef762 2024-05-06T13:37:10.118 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (5m) 3m ago 5m 15.9M - 1dbe0e931976 eb6f92239232 2024-05-06T13:37:10.118 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (5m) 3m ago 5m 16.5M - 1dbe0e931976 001e61e5f922 2024-05-06T13:37:10.118 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (8m) 3m ago 8m 47.1M 4016M 17.2.0 e1d6a67b021e 2ccee2bb6e96 2024-05-06T13:37:10.119 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (8m) 3m ago 8m 51.2M 4016M 17.2.0 e1d6a67b021e 1d9e6609b585 2024-05-06T13:37:10.119 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (7m) 3m ago 7m 46.3M 4016M 17.2.0 e1d6a67b021e 37a61d295111 2024-05-06T13:37:10.119 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (7m) 3m ago 7m 45.0M 4016M 17.2.0 e1d6a67b021e 8feb7c077aaa 2024-05-06T13:37:10.119 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (6m) 3m ago 6m 47.9M 4272M 17.2.0 e1d6a67b021e f865acc44a06 2024-05-06T13:37:10.119 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (6m) 3m ago 6m 44.1M 4272M 17.2.0 e1d6a67b021e a55bd4122f6f 2024-05-06T13:37:10.119 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (6m) 3m ago 6m 42.9M 4272M 17.2.0 e1d6a67b021e 362938a0b357 2024-05-06T13:37:10.119 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (5m) 3m ago 5m 45.2M 4272M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T13:37:10.119 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (3m) 3m ago 5m 45.6M - 514e6a882f6e 488818ee0077 2024-05-06T13:37:10.119 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (3m) 3m ago 3m 70.3M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T13:37:10.119 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (3m) 3m ago 3m 69.2M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T13:37:10.722 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'sleep 180' 2024-05-06T13:37:10.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:10 smithi077 ceph-mon[22428]: from='client.24916 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:37:10.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:10 smithi077 ceph-mon[22428]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:10.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:10 smithi077 ceph-mon[27166]: from='client.24916 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:37:10.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:10 smithi077 ceph-mon[27166]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:10.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:10 smithi170 ceph-mon[23272]: from='client.24916 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:37:10.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:10 smithi170 ceph-mon[23272]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:12.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:12 smithi077 ceph-mon[22428]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:12.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:12 smithi077 ceph-mon[27166]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:12.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:12 smithi170 ceph-mon[23272]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:14.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:14 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:37:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:37:14.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:14 smithi170 ceph-mon[23272]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:15.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:14 smithi077 ceph-mon[22428]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:15.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:37:14 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:37:14] "GET /metrics HTTP/1.1" 200 207962 "" "Prometheus/2.33.4" 2024-05-06T13:37:15.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:14 smithi077 ceph-mon[27166]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:16.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:16 smithi170 ceph-mon[23272]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:17.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:16 smithi077 ceph-mon[22428]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:17.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:16 smithi077 ceph-mon[27166]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:18.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:18 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:37:19.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:18 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:37:19.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:18 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:37:19.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:19 smithi170 ceph-mon[23272]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:20.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:19 smithi077 ceph-mon[22428]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:20.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:19 smithi077 ceph-mon[27166]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:21.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:21 smithi170 ceph-mon[23272]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:22.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:21 smithi077 ceph-mon[22428]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:22.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:21 smithi077 ceph-mon[27166]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:23.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:23 smithi170 ceph-mon[23272]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:24.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:23 smithi077 ceph-mon[22428]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:24.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:23 smithi077 ceph-mon[27166]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:24.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:37:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:37:24.953 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:37:24 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:37:24] "GET /metrics HTTP/1.1" 200 207962 "" "Prometheus/2.33.4" 2024-05-06T13:37:25.973 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:25 smithi077 ceph-mon[22428]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:25.973 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:25 smithi077 ceph-mon[27166]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:26.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:25 smithi170 ceph-mon[23272]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:28.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:27 smithi077 ceph-mon[22428]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:28.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:27 smithi077 ceph-mon[27166]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:28.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:27 smithi170 ceph-mon[23272]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:29.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:28 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:37:29.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:28 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:37:29.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:28 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:37:30.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:29 smithi077 ceph-mon[22428]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:37:30.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:29 smithi077 ceph-mon[27166]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:37:30.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:29 smithi170 ceph-mon[23272]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:37:32.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:31 smithi077 ceph-mon[22428]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:32.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:31 smithi077 ceph-mon[27166]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:32.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:31 smithi170 ceph-mon[23272]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:34.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:33 smithi077 ceph-mon[22428]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:37:34.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:33 smithi077 ceph-mon[27166]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:37:34.057 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:33 smithi170 ceph-mon[23272]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:37:34.312 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:34 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[24449]: ::ffff:172.21.15.170 - - [06/May/2024:13:37:34] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:37:35.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:37:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:37:34] "GET /metrics HTTP/1.1" 200 207956 "" "Prometheus/2.33.4" 2024-05-06T13:37:35.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:35 smithi170 systemd[1]: Stopping Ceph mgr.x for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:37:35.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:35 smithi170 bash[42894]: Error: no container with name or ID "ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr.x" found: no such container 2024-05-06T13:37:35.918 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:35 smithi170 ceph-mon[23272]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:37:35.918 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:35 smithi170 podman[42907]: 2024-05-06 13:37:35.654509612 +0000 UTC m=+0.136200605 container died 82c841abfafd7ae48579ff72a3c2353a22dda83f9e02ba8dccb1bbd5b7ab983b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x, distribution-scope=public, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, version=8, name=centos-stream, io.openshift.expose-services=, GIT_CLEAN=True) 2024-05-06T13:37:36.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:35 smithi077 ceph-mon[22428]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:37:36.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:35 smithi077 ceph-mon[27166]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:37:36.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:36 smithi170 podman[42907]: 2024-05-06 13:37:36.016894178 +0000 UTC m=+0.498585161 container cleanup 82c841abfafd7ae48579ff72a3c2353a22dda83f9e02ba8dccb1bbd5b7ab983b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x, vcs-type=git, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, version=8, RELEASE=HEAD, release=754, maintainer=Guillaume Abrioux ) 2024-05-06T13:37:36.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:36 smithi170 bash[42907]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x 2024-05-06T13:37:36.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:36 smithi170 bash[42934]: Error: no container with name or ID "ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr.x" found: no such container 2024-05-06T13:37:36.760 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:36 smithi170 podman[42921]: 2024-05-06 13:37:36.335161313 +0000 UTC m=+0.677880102 container remove 82c841abfafd7ae48579ff72a3c2353a22dda83f9e02ba8dccb1bbd5b7ab983b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x, CEPH_POINT_RELEASE=-17.2.0, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, version=8, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, RELEASE=HEAD, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, io.openshift.tags=base centos centos-stream) 2024-05-06T13:37:36.760 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:36 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-05-06T13:37:36.760 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:36 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-05-06T13:37:36.760 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:36 smithi170 systemd[1]: Stopped Ceph mgr.x for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:37:36.761 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:36 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mgr.x.service: Consumed 17.064s CPU time. 2024-05-06T13:37:36.761 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:36 smithi170 systemd[1]: Starting Ceph mgr.x for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:37:37.047 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:36 smithi170 podman[43037]: 2024-05-06 13:37:36.663946564 +0000 UTC m=+0.018567016 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:37:37.047 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:36 smithi170 podman[43037]: 2024-05-06 13:37:36.773453586 +0000 UTC m=+0.128074031 container create 63ef6825ff68ea8352f7e7a157677daef5c84d1144b93dcc93ada99c06d81bae (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x, RELEASE=squid-70ce953, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T13:37:37.047 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:36 smithi170 podman[43037]: 2024-05-06 13:37:36.964136075 +0000 UTC m=+0.318756515 container init 63ef6825ff68ea8352f7e7a157677daef5c84d1144b93dcc93ada99c06d81bae (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T13:37:37.047 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:36 smithi170 podman[43037]: 2024-05-06 13:37:36.966793628 +0000 UTC m=+0.321414077 container start 63ef6825ff68ea8352f7e7a157677daef5c84d1144b93dcc93ada99c06d81bae (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-70ce953, io.buildah.version=1.33.5) 2024-05-06T13:37:37.047 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:36 smithi170 ceph-mgr[43056]: -- 172.21.15.170:0/2923423884 <== mon.2 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d73bfde5a0 con 0x55d73bfb3000 2024-05-06T13:37:37.047 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:37 smithi170 bash[43037]: 63ef6825ff68ea8352f7e7a157677daef5c84d1144b93dcc93ada99c06d81bae 2024-05-06T13:37:37.297 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:37 smithi170 systemd[1]: Started Ceph mgr.x for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:37:37.297 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:37 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:37:37.121+0000 7f89aac17200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T13:37:37.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:37 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:37:37.320+0000 7f89aac17200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:37:37.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:37 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:37:37.445+0000 7f89aac17200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T13:37:38.076 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:37 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:37:37.808+0000 7f89aac17200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T13:37:38.076 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:37 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:37:37.902+0000 7f89aac17200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T13:37:38.301 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:38 smithi077 ceph-mon[27166]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:38.301 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:38 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:37:38.301 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:38 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:37:38.301 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:38 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:37:38.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:38 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:37:38.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:38 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:37:38.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:38 smithi077 ceph-mon[22428]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:38.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:38 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:37:38.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:38 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:37:38.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:38 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:37:38.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:38 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:37:38.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:38 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:37:38.538 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:38 smithi170 ceph-mon[23272]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:38.538 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:38 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:37:38.538 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:38 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:37:38.538 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:38 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:37:38.538 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:38 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:37:38.538 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:38 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:37:39.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:39 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:37:39.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:39 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:37:39.379 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:39 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:37:39.379 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:39 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:37:39.199+0000 7f89aac17200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T13:37:39.826 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:39 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:37:39.378+0000 7f89aac17200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T13:37:39.826 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:39 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:37:39.552+0000 7f89aac17200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T13:37:40.110 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:39 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:37:39.859+0000 7f89aac17200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T13:37:40.110 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:39 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:37:39.953+0000 7f89aac17200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T13:37:40.417 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:40 smithi077 ceph-mon[22428]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:40.417 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:40 smithi077 ceph-mon[27166]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:40.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:40 smithi170 ceph-mon[23272]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:40.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:40 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:37:40.161+0000 7f89aac17200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T13:37:40.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:40 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:37:40.357+0000 7f89aac17200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:37:41.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:40 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:37:40.983+0000 7f89aac17200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T13:37:41.719 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:41 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:37:41.422+0000 7f89aac17200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T13:37:41.719 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:41 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:37:41.516+0000 7f89aac17200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T13:37:42.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:41 smithi077 ceph-mon[22428]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:42.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:41 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:37:42.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:41 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:37:42.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:41 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:37:42.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:41 smithi077 ceph-mon[27166]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:42.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:41 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:37:42.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:41 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:37:42.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:41 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:37:42.076 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:41 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:37:41.720+0000 7f89aac17200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-06T13:37:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:41 smithi170 ceph-mon[23272]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:41 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:37:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:41 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:37:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:41 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:37:42.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:42 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:37:42.249+0000 7f89aac17200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T13:37:42.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:42 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:37:42.342+0000 7f89aac17200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T13:37:42.934 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:42 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:37:42.586+0000 7f89aac17200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T13:37:42.934 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:42 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:37:42.720+0000 7f89aac17200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T13:37:42.934 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:42 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:37:42.818+0000 7f89aac17200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T13:37:42.934 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:42 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:37:42.933+0000 7f89aac17200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T13:37:43.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:43 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:37:43.029+0000 7f89aac17200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T13:37:44.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:43 smithi077 ceph-mon[22428]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:44.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:43 smithi077 ceph-mon[27166]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:44.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:43 smithi170 ceph-mon[23272]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:44.076 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:43 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:37:43.854+0000 7f89aac17200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T13:37:44.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:44 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:37:44.075+0000 7f89aac17200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T13:37:44.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:44 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:37:44] ENGINE Bus STARTING 2024-05-06T13:37:44.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:44 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: CherryPy Checker: 2024-05-06T13:37:44.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:44 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: The Application mounted at '' has an empty config. 2024-05-06T13:37:44.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:44 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:37:44.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:44 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:37:44] ENGINE Serving on http://:::9283 2024-05-06T13:37:44.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:44 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:37:44] ENGINE Bus STARTED 2024-05-06T13:37:45.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:37:44 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:37:44] "GET /metrics HTTP/1.1" 200 207970 "" "Prometheus/2.33.4" 2024-05-06T13:37:45.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:44 smithi077 ceph-mon[27166]: Standby manager daemon x restarted 2024-05-06T13:37:45.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:44 smithi077 ceph-mon[27166]: Standby manager daemon x started 2024-05-06T13:37:45.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:44 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T13:37:45.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:44 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:37:45.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:44 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T13:37:45.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:44 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:37:45.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:37:45.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:37:45.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:44 smithi077 ceph-mon[22428]: Standby manager daemon x restarted 2024-05-06T13:37:45.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:44 smithi077 ceph-mon[22428]: Standby manager daemon x started 2024-05-06T13:37:45.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:44 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T13:37:45.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:44 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:37:45.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:44 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T13:37:45.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:44 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:37:45.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:37:45.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:37:45.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:44 smithi170 ceph-mon[23272]: Standby manager daemon x restarted 2024-05-06T13:37:45.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:44 smithi170 ceph-mon[23272]: Standby manager daemon x started 2024-05-06T13:37:45.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:44 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T13:37:45.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:44 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:37:45.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:44 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T13:37:45.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:44 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:37:45.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:37:45.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:37:46.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:45 smithi077 ceph-mon[22428]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:46.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:45 smithi077 ceph-mon[22428]: mgrmap e21: y(active, since 6m), standbys: x 2024-05-06T13:37:46.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:45 smithi077 ceph-mon[27166]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:46.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:45 smithi077 ceph-mon[27166]: mgrmap e21: y(active, since 6m), standbys: x 2024-05-06T13:37:46.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:45 smithi170 ceph-mon[23272]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:46.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:45 smithi170 ceph-mon[23272]: mgrmap e21: y(active, since 6m), standbys: x 2024-05-06T13:37:48.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:47 smithi077 ceph-mon[22428]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:48.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:47 smithi077 ceph-mon[27166]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:48.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:47 smithi170 ceph-mon[23272]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:49.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:48 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:37:49.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:48 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:37:49.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:48 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:37:50.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:49 smithi077 ceph-mon[22428]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:50.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:49 smithi077 ceph-mon[27166]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:50.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:49 smithi170 ceph-mon[23272]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:52.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:51 smithi077 ceph-mon[22428]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:52.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:51 smithi077 ceph-mon[27166]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:52.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:51 smithi170 ceph-mon[23272]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:53.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:53 smithi077 ceph-mon[22428]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:53.805 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:53 smithi077 ceph-mon[27166]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:53.806 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:37:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:37:53.507Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs" 2024-05-06T13:37:53.806 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:37:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:37:53.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:37:54.058 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:53 smithi170 ceph-mon[23272]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:54.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:37:54 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:37:54] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:37:54.948 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:37:54 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:37:54] "GET /metrics HTTP/1.1" 200 207970 "" "Prometheus/2.33.4" 2024-05-06T13:37:56.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:55 smithi077 ceph-mon[22428]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:56.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:55 smithi077 ceph-mon[27166]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:56.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:55 smithi170 ceph-mon[23272]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:37:58.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:57 smithi077 ceph-mon[22428]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:58.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:57 smithi077 ceph-mon[27166]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:58.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:57 smithi170 ceph-mon[23272]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:37:59.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:58 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:37:59.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:58 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:37:59.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:58 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:38:00.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:37:59 smithi077 ceph-mon[22428]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:00.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:37:59 smithi077 ceph-mon[27166]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:00.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:37:59 smithi170 ceph-mon[23272]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:02.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:01 smithi077 ceph-mon[22428]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:02.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:01 smithi077 ceph-mon[27166]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:02.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:01 smithi170 ceph-mon[23272]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:03.771 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:38:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=error ts=2024-05-06T13:38:03.506Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs" 2024-05-06T13:38:03.771 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:38:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:38:03.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs" 2024-05-06T13:38:03.771 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:38:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:38:03.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:38:04.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:03 smithi077 ceph-mon[22428]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:04.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:03 smithi077 ceph-mon[27166]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:04.058 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:03 smithi170 ceph-mon[23272]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:04.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:38:04 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:38:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:38:05.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:38:04 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:38:04] "GET /metrics HTTP/1.1" 200 207973 "" "Prometheus/2.33.4" 2024-05-06T13:38:06.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:05 smithi077 ceph-mon[27166]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:06.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:05 smithi077 ceph-mon[22428]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:06.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:05 smithi170 ceph-mon[23272]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:08.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:07 smithi077 ceph-mon[27166]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:08.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:07 smithi077 ceph-mon[22428]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:08.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:07 smithi170 ceph-mon[23272]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:09.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:08 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:38:09.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:08 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:38:09.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:08 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:38:10.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:09 smithi077 ceph-mon[22428]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:10.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:09 smithi077 ceph-mon[27166]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:10.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:09 smithi170 ceph-mon[23272]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:12.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:11 smithi077 ceph-mon[22428]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:12.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:11 smithi077 ceph-mon[27166]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:12.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:11 smithi170 ceph-mon[23272]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:13.796 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:38:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=error ts=2024-05-06T13:38:13.506Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:38:13.796 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:38:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:38:13.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs" 2024-05-06T13:38:13.796 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:38:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:38:13.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:38:14.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:13 smithi077 ceph-mon[22428]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:14.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:13 smithi077 ceph-mon[27166]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:14.058 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:13 smithi170 ceph-mon[23272]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:14.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:38:14 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:38:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:38:15.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:38:14 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:38:14] "GET /metrics HTTP/1.1" 200 207978 "" "Prometheus/2.33.4" 2024-05-06T13:38:16.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:15 smithi077 ceph-mon[22428]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:16.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:15 smithi077 ceph-mon[27166]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:16.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:15 smithi170 ceph-mon[23272]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:18.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:17 smithi170 ceph-mon[23272]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:17 smithi077 ceph-mon[22428]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:18.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:17 smithi077 ceph-mon[27166]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:19.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:18 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:38:19.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:18 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:38:19.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:18 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:38:20.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:19 smithi170 ceph-mon[23272]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:20.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:19 smithi077 ceph-mon[22428]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:20.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:19 smithi077 ceph-mon[27166]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:22.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:21 smithi170 ceph-mon[23272]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:22.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:21 smithi077 ceph-mon[22428]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:22.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:21 smithi077 ceph-mon[27166]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:23.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:38:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=error ts=2024-05-06T13:38:23.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs" 2024-05-06T13:38:23.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:38:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:38:23.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs" 2024-05-06T13:38:23.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:38:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:38:23.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:38:24.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:23 smithi170 ceph-mon[23272]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:24.076 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:38:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:38:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:38:24.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:23 smithi077 ceph-mon[22428]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:24.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:23 smithi077 ceph-mon[27166]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:25.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:38:24 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:38:24] "GET /metrics HTTP/1.1" 200 207978 "" "Prometheus/2.33.4" 2024-05-06T13:38:26.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:25 smithi170 ceph-mon[23272]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:26.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:25 smithi077 ceph-mon[27166]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:26.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:25 smithi077 ceph-mon[22428]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:28.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:27 smithi170 ceph-mon[23272]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:28.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:27 smithi077 ceph-mon[22428]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:28.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:27 smithi077 ceph-mon[27166]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:29.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:28 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:38:29.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:28 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:38:29.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:28 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:38:30.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:29 smithi077 ceph-mon[22428]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:30.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:29 smithi077 ceph-mon[27166]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:30.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:29 smithi170 ceph-mon[23272]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:32.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:31 smithi077 ceph-mon[22428]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:32.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:31 smithi077 ceph-mon[27166]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:32.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:31 smithi170 ceph-mon[23272]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:33.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:38:33 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=error ts=2024-05-06T13:38:33.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:38:33.803 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:38:33 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:38:33.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:38:33.803 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:38:33 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:38:33.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs" 2024-05-06T13:38:34.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:33 smithi077 ceph-mon[22428]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:34.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:33 smithi077 ceph-mon[27166]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:34.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:33 smithi170 ceph-mon[23272]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:34.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:38:34 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:38:34] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:38:35.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:38:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:38:34] "GET /metrics HTTP/1.1" 200 207972 "" "Prometheus/2.33.4" 2024-05-06T13:38:36.128 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:35 smithi170 ceph-mon[23272]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:36.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:35 smithi077 ceph-mon[22428]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:36.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:35 smithi077 ceph-mon[27166]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:38.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:37 smithi077 ceph-mon[22428]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:38.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:37 smithi077 ceph-mon[27166]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:38.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:37 smithi170 ceph-mon[23272]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:39.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:38 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:38:39.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:38 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:38:39.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:38 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:38:40.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:39 smithi077 ceph-mon[22428]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:40.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:39 smithi077 ceph-mon[27166]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:40.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:39 smithi170 ceph-mon[23272]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:42.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:41 smithi077 ceph-mon[22428]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:42.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:41 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:38:42.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:41 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:38:42.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:41 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:38:42.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:41 smithi077 ceph-mon[27166]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:42.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:41 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:38:42.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:41 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:38:42.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:41 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:38:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:41 smithi170 ceph-mon[23272]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:41 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:38:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:41 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:38:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:41 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:38:43.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:38:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=error ts=2024-05-06T13:38:43.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:38:43.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:38:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:38:43.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:38:43.803 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:38:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:38:43.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs" 2024-05-06T13:38:44.058 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:43 smithi170 ceph-mon[23272]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:44.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:43 smithi077 ceph-mon[22428]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:44.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:43 smithi077 ceph-mon[27166]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:44.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:38:44 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:38:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:38:45.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:38:45.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:44 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:38:45.053 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:38:44 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:38:44] "GET /metrics HTTP/1.1" 200 207978 "" "Prometheus/2.33.4" 2024-05-06T13:38:45.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:38:45.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:44 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:38:45.165 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:38:45.165 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:44 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:38:46.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:45 smithi077 ceph-mon[22428]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:46.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:45 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:38:46.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:45 smithi077 ceph-mon[27166]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:46.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:45 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:38:46.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:45 smithi170 ceph-mon[23272]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:46.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:45 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:38:47.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:46 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:38:47.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:46 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:38:47.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:46 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:38:48.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:47 smithi077 ceph-mon[22428]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:48.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:47 smithi077 ceph-mon[27166]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:48.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:47 smithi170 ceph-mon[23272]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:49.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:48 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:38:49.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:48 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:38:49.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:48 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:38:50.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:49 smithi077 ceph-mon[22428]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:50.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:49 smithi077 ceph-mon[27166]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:50.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:49 smithi170 ceph-mon[23272]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:52.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:51 smithi077 ceph-mon[27166]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:52.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:51 smithi077 ceph-mon[22428]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:52.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:51 smithi170 ceph-mon[23272]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:53.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:38:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=error ts=2024-05-06T13:38:53.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:38:53.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:38:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:38:53.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs" 2024-05-06T13:38:53.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:38:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:38:53.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:38:54.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:53 smithi077 ceph-mon[22428]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:54.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:53 smithi077 ceph-mon[27166]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:54.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:53 smithi170 ceph-mon[23272]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:54.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:38:54 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:38:54] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:38:55.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:38:54 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:38:54] "GET /metrics HTTP/1.1" 200 207978 "" "Prometheus/2.33.4" 2024-05-06T13:38:56.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:55 smithi077 ceph-mon[22428]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:56.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:55 smithi077 ceph-mon[27166]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:56.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:55 smithi170 ceph-mon[23272]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:38:58.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:57 smithi077 ceph-mon[22428]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:58.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:57 smithi077 ceph-mon[27166]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:58.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:57 smithi170 ceph-mon[23272]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:38:59.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:58 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:38:59.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:58 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:38:59.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:58 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:39:00.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:38:59 smithi077 ceph-mon[22428]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:00.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:38:59 smithi077 ceph-mon[27166]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:00.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:38:59 smithi170 ceph-mon[23272]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:02.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:01 smithi077 ceph-mon[22428]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:02.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:01 smithi077 ceph-mon[27166]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:02.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:01 smithi170 ceph-mon[23272]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:03.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:39:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=error ts=2024-05-06T13:39:03.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs" 2024-05-06T13:39:03.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:39:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:39:03.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs" 2024-05-06T13:39:03.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:39:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:39:03.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:39:04.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:03 smithi077 ceph-mon[22428]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:04.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:03 smithi077 ceph-mon[27166]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:04.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:03 smithi170 ceph-mon[23272]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:04.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:39:04 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:39:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:39:05.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:39:04 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:39:04] "GET /metrics HTTP/1.1" 200 207979 "" "Prometheus/2.33.4" 2024-05-06T13:39:06.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:05 smithi077 ceph-mon[22428]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:06.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:05 smithi077 ceph-mon[27166]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:06.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:05 smithi170 ceph-mon[23272]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:08.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:07 smithi077 ceph-mon[22428]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:08.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:07 smithi077 ceph-mon[27166]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:08.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:07 smithi170 ceph-mon[23272]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:09.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:08 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:39:09.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:08 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:39:09.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:08 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:39:10.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:09 smithi077 ceph-mon[22428]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:10.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:09 smithi077 ceph-mon[27166]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:10.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:09 smithi170 ceph-mon[23272]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:12.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:11 smithi077 ceph-mon[22428]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:12.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:11 smithi077 ceph-mon[27166]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:12.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:11 smithi170 ceph-mon[23272]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:13.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:39:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=error ts=2024-05-06T13:39:13.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs" 2024-05-06T13:39:13.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:39:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:39:13.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs" 2024-05-06T13:39:13.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:39:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:39:13.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:39:14.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:13 smithi077 ceph-mon[22428]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:14.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:13 smithi077 ceph-mon[27166]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:14.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:13 smithi170 ceph-mon[23272]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:14.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:39:14 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:39:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:39:15.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:39:14 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:39:14] "GET /metrics HTTP/1.1" 200 207981 "" "Prometheus/2.33.4" 2024-05-06T13:39:16.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:15 smithi077 ceph-mon[22428]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:16.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:15 smithi077 ceph-mon[27166]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:16.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:15 smithi170 ceph-mon[23272]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:17 smithi077 ceph-mon[22428]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:18.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:17 smithi077 ceph-mon[27166]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:18.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:17 smithi170 ceph-mon[23272]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:19.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:18 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:39:19.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:18 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:39:19.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:18 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:39:20.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:19 smithi077 ceph-mon[22428]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:20.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:19 smithi077 ceph-mon[27166]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:20.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:19 smithi170 ceph-mon[23272]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:22.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:21 smithi077 ceph-mon[22428]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:22.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:21 smithi077 ceph-mon[27166]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:22.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:21 smithi170 ceph-mon[23272]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:23.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:39:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=error ts=2024-05-06T13:39:23.513Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:39:23.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:39:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:39:23.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs" 2024-05-06T13:39:23.803 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:39:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:39:23.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:39:24.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:23 smithi077 ceph-mon[22428]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:24.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:23 smithi077 ceph-mon[27166]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:24.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:23 smithi170 ceph-mon[23272]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:24.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:39:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:39:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:39:25.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:39:24 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:39:24] "GET /metrics HTTP/1.1" 200 207981 "" "Prometheus/2.33.4" 2024-05-06T13:39:26.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:25 smithi077 ceph-mon[22428]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:26.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:25 smithi077 ceph-mon[27166]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:26.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:25 smithi170 ceph-mon[23272]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:28.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:27 smithi077 ceph-mon[22428]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:28.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:27 smithi077 ceph-mon[27166]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:28.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:27 smithi170 ceph-mon[23272]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:29.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:29 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:39:29.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:29 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:39:29.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:29 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:39:30.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:30 smithi077 ceph-mon[22428]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:30.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:30 smithi077 ceph-mon[27166]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:30.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:30 smithi170 ceph-mon[23272]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:32.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:32 smithi077 ceph-mon[22428]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:32.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:32 smithi077 ceph-mon[27166]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:32.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:32 smithi170 ceph-mon[23272]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:33.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:39:33 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=error ts=2024-05-06T13:39:33.513Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:39:33.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:39:33 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:39:33.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs" 2024-05-06T13:39:33.803 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:39:33 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:39:33.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:39:34.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:34 smithi077 ceph-mon[22428]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:34.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:34 smithi077 ceph-mon[27166]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:34.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:34 smithi170 ceph-mon[23272]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:34.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:39:34 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:39:34] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:39:35.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:39:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:39:34] "GET /metrics HTTP/1.1" 200 207978 "" "Prometheus/2.33.4" 2024-05-06T13:39:36.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:36 smithi077 ceph-mon[22428]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:36.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:36 smithi077 ceph-mon[27166]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:36.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:36 smithi170 ceph-mon[23272]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:38.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:38 smithi077 ceph-mon[27166]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:38.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:38 smithi077 ceph-mon[22428]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:38.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:38 smithi170 ceph-mon[23272]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:39.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:39 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:39:39.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:39 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:39:39.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:39 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:39:40.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:40 smithi077 ceph-mon[22428]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:40.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:40 smithi077 ceph-mon[27166]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:40.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:40 smithi170 ceph-mon[23272]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:42.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:42 smithi077 ceph-mon[22428]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:42.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:42 smithi077 ceph-mon[27166]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:42.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:42 smithi170 ceph-mon[23272]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:43.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:39:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=error ts=2024-05-06T13:39:43.514Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:39:43.803 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:39:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:39:43.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs" 2024-05-06T13:39:43.803 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:39:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:39:43.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:39:44.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:44 smithi077 ceph-mon[22428]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:44.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:44 smithi077 ceph-mon[27166]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:44.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:44 smithi170 ceph-mon[23272]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:44.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:39:44 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:39:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:39:45.037 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:39:44 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:39:44] "GET /metrics HTTP/1.1" 200 207981 "" "Prometheus/2.33.4" 2024-05-06T13:39:45.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:45 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:39:45.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:45 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:39:45.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:45 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:39:45.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:45 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:39:45.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:45 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:39:45.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:45 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:39:46.218 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:46 smithi077 ceph-mon[22428]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:46.218 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:46 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:39:46.218 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:46 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:39:46.218 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:46 smithi077 ceph-mon[27166]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:46.218 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:46 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:39:46.218 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:46 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:39:46.219 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:46 smithi170 ceph-mon[23272]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:46.219 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:46 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:39:46.219 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:46 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:39:47.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:47 smithi077 ceph-mon[22428]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:39:47.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:47 smithi077 ceph-mon[27166]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:39:47.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:47 smithi170 ceph-mon[23272]: from='mgr.24352 172.21.15.77:0/2989050448' entity='mgr.y' 2024-05-06T13:39:48.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:48 smithi077 ceph-mon[22428]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:48.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:48 smithi077 ceph-mon[27166]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:48.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:48 smithi170 ceph-mon[23272]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:49.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:49 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:39:49.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:49 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:39:49.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:49 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:39:50.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:50 smithi077 ceph-mon[22428]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:50.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:50 smithi077 ceph-mon[27166]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:50.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:50 smithi170 ceph-mon[23272]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:52.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:52 smithi077 ceph-mon[22428]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:52.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:52 smithi077 ceph-mon[27166]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:52.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:52 smithi170 ceph-mon[23272]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:53.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:39:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=error ts=2024-05-06T13:39:53.515Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs" 2024-05-06T13:39:53.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:39:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:39:53.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs" 2024-05-06T13:39:53.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:39:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:39:53.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:39:54.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:54 smithi170 ceph-mon[23272]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:54.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:39:54 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:39:54] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:39:54.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:54 smithi077 ceph-mon[22428]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:54.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:54 smithi077 ceph-mon[27166]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:55.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:39:54 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:39:54] "GET /metrics HTTP/1.1" 200 207981 "" "Prometheus/2.33.4" 2024-05-06T13:39:56.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:56 smithi077 ceph-mon[27166]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:56.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:56 smithi077 ceph-mon[22428]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:56.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:56 smithi170 ceph-mon[23272]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:39:58.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:58 smithi077 ceph-mon[22428]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:58.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:58 smithi077 ceph-mon[27166]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:58.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:58 smithi170 ceph-mon[23272]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:39:59.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:39:59 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:39:59.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:39:59 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:39:59.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:39:59 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:40:00.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:00 smithi077 ceph-mon[22428]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:40:00.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:00 smithi077 ceph-mon[22428]: overall HEALTH_OK 2024-05-06T13:40:00.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:00 smithi077 ceph-mon[27166]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:40:00.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:00 smithi077 ceph-mon[27166]: overall HEALTH_OK 2024-05-06T13:40:00.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:00 smithi170 ceph-mon[23272]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:40:00.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:00 smithi170 ceph-mon[23272]: overall HEALTH_OK 2024-05-06T13:40:02.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:02 smithi077 ceph-mon[22428]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:40:02.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:02 smithi077 ceph-mon[27166]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:40:02.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:02 smithi170 ceph-mon[23272]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:40:03.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=error ts=2024-05-06T13:40:03.515Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:40:03.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:40:03.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs" 2024-05-06T13:40:03.803 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:40:03.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:40:04.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:04 smithi170 ceph-mon[23272]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:40:04.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:04 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:40:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:40:04.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:04 smithi077 ceph-mon[22428]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:40:04.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:04 smithi077 ceph-mon[27166]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:40:05.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:04 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:40:04] "GET /metrics HTTP/1.1" 200 207976 "" "Prometheus/2.33.4" 2024-05-06T13:40:06.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:06 smithi077 ceph-mon[22428]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:40:06.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:06 smithi077 ceph-mon[27166]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:40:06.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:06 smithi170 ceph-mon[23272]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:40:08.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:08 smithi077 ceph-mon[22428]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:40:08.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:08 smithi077 ceph-mon[27166]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:40:08.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:08 smithi170 ceph-mon[23272]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:40:09.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:09 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:40:09.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:09 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:40:09.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:09 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:40:10.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:10 smithi077 ceph-mon[22428]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:40:10.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:10 smithi077 ceph-mon[27166]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:40:10.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:10 smithi170 ceph-mon[23272]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:40:12.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:12 smithi077 ceph-mon[22428]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:40:12.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:12 smithi077 ceph-mon[27166]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:40:12.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:12 smithi170 ceph-mon[23272]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:40:13.252 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch ps' 2024-05-06T13:40:13.784 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=error ts=2024-05-06T13:40:13.515Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs" 2024-05-06T13:40:13.784 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:40:13.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs" 2024-05-06T13:40:13.784 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:40:13.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:40:14.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:14 smithi170 ceph-mon[23272]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-05-06T13:40:14.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:14 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:40:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:40:14.535 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:14 smithi077 ceph-mon[22428]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-05-06T13:40:14.536 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:14 smithi077 ceph-mon[27166]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-05-06T13:40:14.802 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:14 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:40:14] "GET /metrics HTTP/1.1" 200 207975 "" "Prometheus/2.33.4" 2024-05-06T13:40:15.118 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T13:40:15.118 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (6m) 2m ago 7m 27.2M - ba2b418f427c d17996419263 2024-05-06T13:40:15.118 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (7m) 89s ago 7m 55.7M - 8.3.5 dad864ee21e9 4b872c67236d 2024-05-06T13:40:15.118 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (6m) 2m ago 6m 53.3M - 3.5 e1d6a67b021e d705e2e893a0 2024-05-06T13:40:15.119 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283 running (2m) 89s ago 11m 452M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 63ef6825ff68 2024-05-06T13:40:15.119 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:9283 running (13m) 2m ago 13m 459M - 17.2.0 e1d6a67b021e 79a55d163d05 2024-05-06T13:40:15.119 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (13m) 2m ago 13m 46.5M 2048M 17.2.0 e1d6a67b021e ac46f91694f8 2024-05-06T13:40:15.119 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (12m) 89s ago 12m 40.2M 2048M 17.2.0 e1d6a67b021e 24be6fce9d28 2024-05-06T13:40:15.119 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (12m) 2m ago 12m 36.8M 2048M 17.2.0 e1d6a67b021e 83af336ef762 2024-05-06T13:40:15.119 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (8m) 2m ago 8m 15.9M - 1dbe0e931976 eb6f92239232 2024-05-06T13:40:15.119 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (8m) 89s ago 8m 19.5M - 1dbe0e931976 001e61e5f922 2024-05-06T13:40:15.119 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (11m) 2m ago 11m 49.5M 4016M 17.2.0 e1d6a67b021e 2ccee2bb6e96 2024-05-06T13:40:15.119 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (11m) 2m ago 11m 52.8M 4016M 17.2.0 e1d6a67b021e 1d9e6609b585 2024-05-06T13:40:15.119 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (10m) 2m ago 10m 48.8M 4016M 17.2.0 e1d6a67b021e 37a61d295111 2024-05-06T13:40:15.119 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (10m) 2m ago 10m 48.2M 4016M 17.2.0 e1d6a67b021e 8feb7c077aaa 2024-05-06T13:40:15.119 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (10m) 89s ago 10m 51.5M 4272M 17.2.0 e1d6a67b021e f865acc44a06 2024-05-06T13:40:15.119 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (9m) 89s ago 9m 47.9M 4272M 17.2.0 e1d6a67b021e a55bd4122f6f 2024-05-06T13:40:15.119 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (9m) 89s ago 9m 46.2M 4272M 17.2.0 e1d6a67b021e 362938a0b357 2024-05-06T13:40:15.119 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (8m) 89s ago 8m 48.8M 4272M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T13:40:15.120 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (6m) 89s ago 8m 68.9M - 514e6a882f6e 488818ee0077 2024-05-06T13:40:15.120 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (6m) 2m ago 6m 71.4M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T13:40:15.120 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (6m) 89s ago 6m 70.8M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T13:40:15.711 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph versions' 2024-05-06T13:40:16.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:16 smithi077 ceph-mon[22428]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-05-06T13:40:16.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:16 smithi077 ceph-mon[22428]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:40:16.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:16 smithi077 ceph-mon[27166]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-05-06T13:40:16.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:16 smithi077 ceph-mon[27166]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:40:16.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:16 smithi170 ceph-mon[23272]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-05-06T13:40:16.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:16 smithi170 ceph-mon[23272]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:40:17.660 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:40:17.661 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-05-06T13:40:17.661 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-05-06T13:40:17.661 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:40:17.661 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-05-06T13:40:17.661 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-05-06T13:40:17.661 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 1 2024-05-06T13:40:17.661 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:40:17.661 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-05-06T13:40:17.661 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-06T13:40:17.661 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:40:17.661 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": {}, 2024-05-06T13:40:17.661 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw": { 2024-05-06T13:40:17.661 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T13:40:17.661 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:40:17.661 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-05-06T13:40:17.662 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14, 2024-05-06T13:40:17.662 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 1 2024-05-06T13:40:17.662 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T13:40:17.662 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:40:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:18 smithi077 ceph-mon[22428]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:40:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:18 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2174324149' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:40:18.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:18 smithi077 ceph-mon[27166]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:40:18.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:18 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2174324149' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:40:18.332 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph -s' 2024-05-06T13:40:18.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:18 smithi170 ceph-mon[23272]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:40:18.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:18 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2174324149' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:40:19.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:19 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:40:19.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:19 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:40:19.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:19 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:40:20.411 INFO:teuthology.orchestra.run.smithi077.stdout: cluster: 2024-05-06T13:40:20.411 INFO:teuthology.orchestra.run.smithi077.stdout: id: 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:40:20.411 INFO:teuthology.orchestra.run.smithi077.stdout: health: HEALTH_OK 2024-05-06T13:40:20.411 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:40:20.411 INFO:teuthology.orchestra.run.smithi077.stdout: services: 2024-05-06T13:40:20.411 INFO:teuthology.orchestra.run.smithi077.stdout: mon: 3 daemons, quorum a,c,b (age 12m) 2024-05-06T13:40:20.411 INFO:teuthology.orchestra.run.smithi077.stdout: mgr: y(active, since 8m), standbys: x 2024-05-06T13:40:20.411 INFO:teuthology.orchestra.run.smithi077.stdout: osd: 8 osds: 8 up (since 8m), 8 in (since 9m) 2024-05-06T13:40:20.411 INFO:teuthology.orchestra.run.smithi077.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-05-06T13:40:20.411 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:40:20.411 INFO:teuthology.orchestra.run.smithi077.stdout: data: 2024-05-06T13:40:20.411 INFO:teuthology.orchestra.run.smithi077.stdout: pools: 6 pools, 161 pgs 2024-05-06T13:40:20.412 INFO:teuthology.orchestra.run.smithi077.stdout: objects: 209 objects, 585 KiB 2024-05-06T13:40:20.412 INFO:teuthology.orchestra.run.smithi077.stdout: usage: 68 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:40:20.412 INFO:teuthology.orchestra.run.smithi077.stdout: pgs: 161 active+clean 2024-05-06T13:40:20.412 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:40:20.412 INFO:teuthology.orchestra.run.smithi077.stdout: io: 2024-05-06T13:40:20.412 INFO:teuthology.orchestra.run.smithi077.stdout: client: 852 B/s rd, 0 op/s rd, 0 op/s wr 2024-05-06T13:40:20.412 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:40:20.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:20 smithi077 ceph-mon[22428]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-05-06T13:40:20.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:20 smithi077 ceph-mon[27166]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-05-06T13:40:20.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:20 smithi170 ceph-mon[23272]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-05-06T13:40:21.055 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph health detail' 2024-05-06T13:40:21.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:21 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/1585197824' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-06T13:40:21.610 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:21 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/1585197824' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-06T13:40:21.610 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:21 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/1585197824' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-06T13:40:22.566 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:22 smithi077 ceph-mon[22428]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:40:22.566 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:22 smithi077 ceph-mon[27166]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:40:22.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:22 smithi170 ceph-mon[23272]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:40:23.138 INFO:teuthology.orchestra.run.smithi077.stdout:HEALTH_OK 2024-05-06T13:40:23.462 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:23 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3678796229' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T13:40:23.462 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:23 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3678796229' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T13:40:23.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:23 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/3678796229' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T13:40:23.737 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=error ts=2024-05-06T13:40:23.516Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:40:23.737 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:40:23.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs" 2024-05-06T13:40:23.738 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:40:23.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:40:23.762 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 2'"'"'' 2024-05-06T13:40:24.313 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:40:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:40:24.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:24 smithi170 ceph-mon[23272]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-05-06T13:40:24.652 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:24 smithi077 ceph-mon[22428]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-05-06T13:40:24.653 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:24 smithi077 ceph-mon[27166]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-05-06T13:40:24.952 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:24 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:40:24] "GET /metrics HTTP/1.1" 200 207975 "" "Prometheus/2.33.4" 2024-05-06T13:40:25.735 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T13:40:26.267 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph mgr fail' 2024-05-06T13:40:26.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:26 smithi077 ceph-mon[22428]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:40:26.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:26 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3647841943' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:40:26.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:26 smithi077 ceph-mon[27166]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:40:26.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:26 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3647841943' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:40:26.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:26 smithi170 ceph-mon[23272]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:40:26.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:26 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/3647841943' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:40:28.635 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:28 smithi077 ceph-mon[22428]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:40:28.635 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:28 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2986139622' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-05-06T13:40:28.636 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:28 smithi077 ceph-mon[27166]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:40:28.636 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:28 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2986139622' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-05-06T13:40:28.636 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:28 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:40:28.332+0000 7f6394fb6700 -1 mgr handle_mgr_map I was active but no longer amignoring --setuser ceph since I am not root 2024-05-06T13:40:28.636 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:28 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ignoring --setgroup ceph since I am not root 2024-05-06T13:40:28.636 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:28 smithi077 ceph-mgr[22664]: -- 172.21.15.77:0/755749131 <== mon.0 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ae7f19c340 con 0x55ae7f174c00 2024-05-06T13:40:28.636 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:28 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:40:28.500+0000 7f9f34315000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T13:40:28.636 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:28 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:40:28.634+0000 7f9f34315000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:40:28.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:28 smithi170 ceph-mon[23272]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:40:28.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:28 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2986139622' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-05-06T13:40:28.826 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:40:28] ENGINE Bus STOPPING 2024-05-06T13:40:28.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:40:28] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-06T13:40:28.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:40:28] ENGINE Bus STOPPED 2024-05-06T13:40:28.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:40:28] ENGINE Bus STARTING 2024-05-06T13:40:28.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:40:28] ENGINE Serving on http://:::9283 2024-05-06T13:40:28.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:40:28] ENGINE Bus STARTED 2024-05-06T13:40:29.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:28 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:40:28.735+0000 7f9f34315000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T13:40:29.052 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:28 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:40:28.803Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.77:8443//api/prometheus_receiver\": dial tcp 172.21.15.77:8443: connect: connection refused" 2024-05-06T13:40:29.100 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'sleep 180' 2024-05-06T13:40:29.338 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:29 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:40:29.059+0000 7f9f34315000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T13:40:29.339 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:29 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:40:29.148+0000 7f9f34315000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T13:40:29.594 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: osdmap e96: 8 total, 8 up, 8 in 2024-05-06T13:40:29.594 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2986139622' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-05-06T13:40:29.594 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: mgrmap e22: x(active, starting, since 0.0165696s) 2024-05-06T13:40:29.594 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:40:29.594 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:40:29.594 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:40:29.594 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T13:40:29.594 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:40:29.594 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:40:29.595 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:40:29.595 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:40:29.595 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:40:29.595 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:40:29.595 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:40:29.595 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:40:29.595 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T13:40:29.595 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T13:40:29.595 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T13:40:29.595 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: Manager daemon x is now available 2024-05-06T13:40:29.595 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:29.595 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: Queued rgw.foo for migration 2024-05-06T13:40:29.595 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-05-06T13:40:29.595 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:29.596 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:40:29.596 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:40:29.596 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:40:29.596 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-06T13:40:29.596 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-06T13:40:29.596 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: Deploying cephadm binary to smithi077 2024-05-06T13:40:29.596 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: Deploying cephadm binary to smithi170 2024-05-06T13:40:29.596 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:29.596 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:29.596 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: osdmap e96: 8 total, 8 up, 8 in 2024-05-06T13:40:29.596 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2986139622' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-05-06T13:40:29.596 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: mgrmap e22: x(active, starting, since 0.0165696s) 2024-05-06T13:40:29.597 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:40:29.597 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:40:29.597 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:40:29.597 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T13:40:29.597 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:40:29.597 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:40:29.597 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:40:29.597 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:40:29.597 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:40:29.597 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:40:29.597 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:40:29.597 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:40:29.597 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T13:40:29.597 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T13:40:29.598 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T13:40:29.598 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: Manager daemon x is now available 2024-05-06T13:40:29.598 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:29.598 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: Queued rgw.foo for migration 2024-05-06T13:40:29.598 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-05-06T13:40:29.598 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:29.598 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:40:29.598 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:40:29.598 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:40:29.598 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-06T13:40:29.598 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-06T13:40:29.598 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: Deploying cephadm binary to smithi077 2024-05-06T13:40:29.598 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: Deploying cephadm binary to smithi170 2024-05-06T13:40:29.598 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:29.599 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:29 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:29.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: osdmap e96: 8 total, 8 up, 8 in 2024-05-06T13:40:29.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2986139622' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-05-06T13:40:29.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: mgrmap e22: x(active, starting, since 0.0165696s) 2024-05-06T13:40:29.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:40:29.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:40:29.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:40:29.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T13:40:29.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:40:29.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:40:29.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:40:29.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:40:29.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:40:29.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:40:29.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:40:29.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:40:29.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T13:40:29.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T13:40:29.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T13:40:29.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: Manager daemon x is now available 2024-05-06T13:40:29.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:29.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: Queued rgw.foo for migration 2024-05-06T13:40:29.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-05-06T13:40:29.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:29.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:40:29.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:40:29.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:40:29.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-06T13:40:29.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-06T13:40:29.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: Deploying cephadm binary to smithi077 2024-05-06T13:40:29.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: Deploying cephadm binary to smithi170 2024-05-06T13:40:29.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:29.829 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:29 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:30.470 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:30 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:40:30.299+0000 7f9f34315000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T13:40:30.470 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:30 smithi077 ceph-mon[22428]: mgrmap e23: x(active, since 1.01974s) 2024-05-06T13:40:30.470 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:30 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:40:30.470 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:30 smithi077 ceph-mon[22428]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:40:30.470 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:30 smithi077 ceph-mon[22428]: [06/May/2024:13:40:29] ENGINE Bus STARTING 2024-05-06T13:40:30.470 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:30 smithi077 ceph-mon[22428]: [06/May/2024:13:40:29] ENGINE Serving on https://172.21.15.170:7150 2024-05-06T13:40:30.470 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:30 smithi077 ceph-mon[22428]: [06/May/2024:13:40:29] ENGINE Serving on http://172.21.15.170:8765 2024-05-06T13:40:30.470 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:30 smithi077 ceph-mon[22428]: [06/May/2024:13:40:29] ENGINE Bus STARTED 2024-05-06T13:40:30.471 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:30 smithi077 ceph-mon[27166]: mgrmap e23: x(active, since 1.01974s) 2024-05-06T13:40:30.471 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:30 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:40:30.471 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:30 smithi077 ceph-mon[27166]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:40:30.471 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:30 smithi077 ceph-mon[27166]: [06/May/2024:13:40:29] ENGINE Bus STARTING 2024-05-06T13:40:30.471 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:30 smithi077 ceph-mon[27166]: [06/May/2024:13:40:29] ENGINE Serving on https://172.21.15.170:7150 2024-05-06T13:40:30.471 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:30 smithi077 ceph-mon[27166]: [06/May/2024:13:40:29] ENGINE Serving on http://172.21.15.170:8765 2024-05-06T13:40:30.471 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:30 smithi077 ceph-mon[27166]: [06/May/2024:13:40:29] ENGINE Bus STARTED 2024-05-06T13:40:30.790 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:30 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:40:30.469+0000 7f9f34315000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T13:40:30.791 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:30 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:40:30.558+0000 7f9f34315000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T13:40:30.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:30 smithi170 ceph-mon[23272]: mgrmap e23: x(active, since 1.01974s) 2024-05-06T13:40:30.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:30 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:40:30.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:30 smithi170 ceph-mon[23272]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:40:30.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:30 smithi170 ceph-mon[23272]: [06/May/2024:13:40:29] ENGINE Bus STARTING 2024-05-06T13:40:30.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:30 smithi170 ceph-mon[23272]: [06/May/2024:13:40:29] ENGINE Serving on https://172.21.15.170:7150 2024-05-06T13:40:30.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:30 smithi170 ceph-mon[23272]: [06/May/2024:13:40:29] ENGINE Serving on http://172.21.15.170:8765 2024-05-06T13:40:30.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:30 smithi170 ceph-mon[23272]: [06/May/2024:13:40:29] ENGINE Bus STARTED 2024-05-06T13:40:31.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:30 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:40:30.887+0000 7f9f34315000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T13:40:31.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:30 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:40:30.981+0000 7f9f34315000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T13:40:31.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:31 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:40:31.076+0000 7f9f34315000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T13:40:31.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:31 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:40:31.318+0000 7f9f34315000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:40:32.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:31 smithi077 ceph-mon[22428]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:40:32.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:31 smithi077 ceph-mon[22428]: mgrmap e24: x(active, since 2s) 2024-05-06T13:40:32.053 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:31 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:40:31.758+0000 7f9f34315000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T13:40:32.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:31 smithi077 ceph-mon[27166]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:40:32.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:31 smithi077 ceph-mon[27166]: mgrmap e24: x(active, since 2s) 2024-05-06T13:40:32.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:31 smithi170 ceph-mon[23272]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:40:32.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:31 smithi170 ceph-mon[23272]: mgrmap e24: x(active, since 2s) 2024-05-06T13:40:32.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:32 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:40:32.249+0000 7f9f34315000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T13:40:32.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:32 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:40:32.345+0000 7f9f34315000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T13:40:33.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:32 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:40:32.644+0000 7f9f34315000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T13:40:33.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:32 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:40:32.733+0000 7f9f34315000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T13:40:33.484 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:33 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:40:33.140+0000 7f9f34315000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T13:40:33.484 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:33 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:40:33.272+0000 7f9f34315000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T13:40:33.484 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:33 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:40:33.366+0000 7f9f34315000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T13:40:33.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:33 smithi077 ceph-mon[22428]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:40:33.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:33 smithi077 ceph-mon[22428]: mgrmap e25: x(active, since 4s) 2024-05-06T13:40:33.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:33 smithi077 ceph-mon[27166]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:40:33.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:33 smithi077 ceph-mon[27166]: mgrmap e25: x(active, since 4s) 2024-05-06T13:40:33.803 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:33 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:40:33.483+0000 7f9f34315000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T13:40:33.803 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:33 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:40:33.574+0000 7f9f34315000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T13:40:33.803 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:33 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=error ts=2024-05-06T13:40:33.517Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.77:8443//api/prometheus_receiver\": dial tcp 172.21.15.77:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:40:33.803 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:33 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:40:33.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.77:8443//api/prometheus_receiver\": dial tcp 172.21.15.77:8443: connect: connection refused" 2024-05-06T13:40:33.803 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:33 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:40:33.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:40:34.057 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:33 smithi170 ceph-mon[23272]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:40:34.058 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:33 smithi170 ceph-mon[23272]: mgrmap e25: x(active, since 4s) 2024-05-06T13:40:34.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:34 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:40:34] "GET /metrics HTTP/1.1" 200 34893 "" "Prometheus/2.33.4" 2024-05-06T13:40:34.432 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:40:34.162+0000 7f9f34315000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T13:40:34.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:34 smithi077 ceph-mon[22428]: Standby manager daemon y started 2024-05-06T13:40:34.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:34 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.77:0/106078965' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-06T13:40:34.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:34 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.77:0/106078965' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:40:34.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:34 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.77:0/106078965' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-06T13:40:34.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:34 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.77:0/106078965' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:40:34.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:34 smithi077 ceph-mon[27166]: Standby manager daemon y started 2024-05-06T13:40:34.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:34 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.77:0/106078965' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-06T13:40:34.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:34 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.77:0/106078965' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:40:34.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:34 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.77:0/106078965' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-06T13:40:34.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:34 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.77:0/106078965' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:40:34.803 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:40:34.533+0000 7f9f34315000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T13:40:34.804 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: [06/May/2024:13:40:34] ENGINE Bus STARTING 2024-05-06T13:40:34.804 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: CherryPy Checker: 2024-05-06T13:40:34.804 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: The Application mounted at '' has an empty config. 2024-05-06T13:40:34.804 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: 2024-05-06T13:40:34.804 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: [06/May/2024:13:40:34] ENGINE Serving on http://:::9283 2024-05-06T13:40:34.804 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: [06/May/2024:13:40:34] ENGINE Bus STARTED 2024-05-06T13:40:34.804 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:40:34] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:40:34.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:34 smithi170 ceph-mon[23272]: Standby manager daemon y started 2024-05-06T13:40:34.831 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:34 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.77:0/106078965' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-06T13:40:34.831 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:34 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.77:0/106078965' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:40:34.831 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:34 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.77:0/106078965' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-06T13:40:34.831 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:34 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.77:0/106078965' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:40:35.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:35 smithi077 ceph-mon[22428]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:40:35.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:35 smithi077 ceph-mon[22428]: mgrmap e26: x(active, since 6s), standbys: y 2024-05-06T13:40:35.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:35 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T13:40:35.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:35 smithi077 ceph-mon[27166]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:40:35.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:35 smithi077 ceph-mon[27166]: mgrmap e26: x(active, since 6s), standbys: y 2024-05-06T13:40:35.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:35 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T13:40:35.803 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:35 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:40:35.535Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=4 err="Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs" 2024-05-06T13:40:35.987 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:35 smithi170 ceph-mon[23272]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:40:35.987 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:35 smithi170 ceph-mon[23272]: mgrmap e26: x(active, since 6s), standbys: y 2024-05-06T13:40:35.987 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:35 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T13:40:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:37 smithi170 ceph-mon[23272]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-05-06T13:40:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:37 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:37.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:37 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:38.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:37 smithi077 ceph-mon[22428]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-05-06T13:40:38.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:37 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:38.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:37 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:38.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:37 smithi077 ceph-mon[27166]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-05-06T13:40:38.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:37 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:38.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:37 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:39.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:39 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:39.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:39 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:39.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:39 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:39.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:39 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:39.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:39 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:39.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:39 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:39.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:39 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:39.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:39 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:39.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:39 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:39.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:39 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:39.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:39 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:39.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:39 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:39.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:39 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:39.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:39 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:39.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:39 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:39.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:39 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:39.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:39 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:39.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:39 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:39.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:39 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:39.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:39 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:39.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:39 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:39.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:39 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:39.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:39 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:39.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:39 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:39.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:39 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:39.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:39 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:39.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:39 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:39.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:39 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:39.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:39 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:39.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:39 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:39.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:39 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:39.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:39 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:39.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:39 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:39.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:39 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:39.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:39 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:39.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:39 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:39.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:39 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:39.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:39 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:39.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:39 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:40.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[22428]: Adjusting osd_memory_target on smithi077 to 2992M 2024-05-06T13:40:40.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[22428]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-06T13:40:40.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:40.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:40.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:40.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:40.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:40.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:40.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:40.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:40.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:40.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:40.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:40.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:40:40.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:40:40.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[27166]: Adjusting osd_memory_target on smithi077 to 2992M 2024-05-06T13:40:40.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[27166]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-06T13:40:40.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:40.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:40.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:40.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:40.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:40.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:40.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:40.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:40.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:40.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:40.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:40.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:40:40.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:40 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:40:40.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:40 smithi170 ceph-mon[23272]: Adjusting osd_memory_target on smithi077 to 2992M 2024-05-06T13:40:40.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:40 smithi170 ceph-mon[23272]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-06T13:40:40.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:40 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:40.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:40 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:40.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:40 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:40.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:40 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:40.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:40 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:40.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:40 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:40.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:40 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:40.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:40 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:40.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:40 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:40.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:40 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-06T13:40:40.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:40 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:40.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:40 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:40:40.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:40 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:40:40.995 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:40 smithi077 systemd[1]: Stopping Ceph node-exporter.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:40:40.995 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:40 smithi077 bash[56253]: Error: no container with name or ID "ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter.a" found: no such container 2024-05-06T13:40:40.995 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:40 smithi077 podman[56266]: 2024-05-06 13:40:40.770913766 +0000 UTC m=+0.132178875 container died eb6f92239232c4dd81c6a40f5707002d95388ca9e2f9c755195fa1af8c575dfc (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T13:40:41.302 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 podman[56266]: 2024-05-06 13:40:41.093865991 +0000 UTC m=+0.455131102 container cleanup eb6f92239232c4dd81c6a40f5707002d95388ca9e2f9c755195fa1af8c575dfc (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T13:40:41.302 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 bash[56266]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a 2024-05-06T13:40:41.302 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 bash[56292]: Error: no container with name or ID "ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter.a" found: no such container 2024-05-06T13:40:41.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[22428]: Adjusting osd_memory_target on smithi170 to 3248M 2024-05-06T13:40:41.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[22428]: Updating smithi077:/etc/ceph/ceph.conf 2024-05-06T13:40:41.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[22428]: Updating smithi170:/etc/ceph/ceph.conf 2024-05-06T13:40:41.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[22428]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:40:41.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[22428]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:40:41.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[22428]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:40:41.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[22428]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:40:41.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[22428]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:40:41.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[22428]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:40:41.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[22428]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:40:41.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:41.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:41.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:41.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:41.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:41.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:41.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:41.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:41.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:41.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[27166]: Adjusting osd_memory_target on smithi170 to 3248M 2024-05-06T13:40:41.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[27166]: Updating smithi077:/etc/ceph/ceph.conf 2024-05-06T13:40:41.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[27166]: Updating smithi170:/etc/ceph/ceph.conf 2024-05-06T13:40:41.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[27166]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:40:41.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[27166]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:40:41.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[27166]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:40:41.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[27166]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:40:41.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[27166]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:40:41.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[27166]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:40:41.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[27166]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:40:41.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:41.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:41.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:41.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:41.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:41.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:41.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:41.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:41.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:41 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:41.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:41 smithi170 ceph-mon[23272]: Adjusting osd_memory_target on smithi170 to 3248M 2024-05-06T13:40:41.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:41 smithi170 ceph-mon[23272]: Updating smithi077:/etc/ceph/ceph.conf 2024-05-06T13:40:41.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:41 smithi170 ceph-mon[23272]: Updating smithi170:/etc/ceph/ceph.conf 2024-05-06T13:40:41.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:41 smithi170 ceph-mon[23272]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:40:41.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:41 smithi170 ceph-mon[23272]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:40:41.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:41 smithi170 ceph-mon[23272]: from='client.14817 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:40:41.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:41 smithi170 ceph-mon[23272]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:40:41.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:41 smithi170 ceph-mon[23272]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:40:41.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:41 smithi170 ceph-mon[23272]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:40:41.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:41 smithi170 ceph-mon[23272]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:40:41.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:41 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:41.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:41 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:41.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:41 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:41.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:41 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:41.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:41 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:41.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:41 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:41.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:41 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:41.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:41 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:41.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:41 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:41.604 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 podman[56280]: 2024-05-06 13:40:41.353958679 +0000 UTC m=+0.587358935 container remove eb6f92239232c4dd81c6a40f5707002d95388ca9e2f9c755195fa1af8c575dfc (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T13:40:41.604 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-05-06T13:40:41.604 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-05-06T13:40:41.605 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 systemd[1]: Stopped Ceph node-exporter.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:40:41.605 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@node-exporter.a.service: Consumed 3.817s CPU time. 2024-05-06T13:40:41.605 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 systemd[1]: Starting Ceph node-exporter.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:40:41.605 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 podman[56361]: 2024-05-06 13:40:41.48081628 +0000 UTC m=+0.017495971 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-05-06T13:40:41.890 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 podman[56361]: 2024-05-06 13:40:41.605045695 +0000 UTC m=+0.141725365 container create 54aee5b22b1c8a81efb1e966694a1742bda9d691f9c9088d9f3dfc6b3907e93b (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T13:40:41.891 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 podman[56361]: 2024-05-06 13:40:41.799083032 +0000 UTC m=+0.335762700 container init 54aee5b22b1c8a81efb1e966694a1742bda9d691f9c9088d9f3dfc6b3907e93b (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T13:40:41.891 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 podman[56361]: 2024-05-06 13:40:41.801971863 +0000 UTC m=+0.338651543 container start 54aee5b22b1c8a81efb1e966694a1742bda9d691f9c9088d9f3dfc6b3907e93b (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T13:40:41.891 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.805Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-05-06T13:40:41.891 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.805Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-05-06T13:40:41.891 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.805Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-05-06T13:40:41.891 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.805Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-05-06T13:40:41.891 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-05-06T13:40:41.891 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=arp 2024-05-06T13:40:41.891 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=bcache 2024-05-06T13:40:41.891 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=bonding 2024-05-06T13:40:41.891 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=btrfs 2024-05-06T13:40:41.891 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=conntrack 2024-05-06T13:40:41.891 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=cpu 2024-05-06T13:40:41.892 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-05-06T13:40:41.892 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=diskstats 2024-05-06T13:40:41.892 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=dmi 2024-05-06T13:40:41.892 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=edac 2024-05-06T13:40:41.892 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=entropy 2024-05-06T13:40:41.892 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-05-06T13:40:41.892 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=filefd 2024-05-06T13:40:41.892 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=filesystem 2024-05-06T13:40:41.892 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=hwmon 2024-05-06T13:40:41.892 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=infiniband 2024-05-06T13:40:41.892 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=ipvs 2024-05-06T13:40:41.892 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=loadavg 2024-05-06T13:40:41.892 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=mdadm 2024-05-06T13:40:41.892 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=meminfo 2024-05-06T13:40:41.893 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=netclass 2024-05-06T13:40:41.893 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=netdev 2024-05-06T13:40:41.893 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=netstat 2024-05-06T13:40:41.893 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=nfs 2024-05-06T13:40:41.893 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=nfsd 2024-05-06T13:40:41.893 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=nvme 2024-05-06T13:40:41.893 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=os 2024-05-06T13:40:41.893 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-05-06T13:40:41.893 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=pressure 2024-05-06T13:40:41.893 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=rapl 2024-05-06T13:40:41.893 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=schedstat 2024-05-06T13:40:41.894 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=sockstat 2024-05-06T13:40:41.894 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=softnet 2024-05-06T13:40:41.894 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=stat 2024-05-06T13:40:41.894 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=tapestats 2024-05-06T13:40:41.894 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=textfile 2024-05-06T13:40:41.894 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-05-06T13:40:41.894 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=time 2024-05-06T13:40:41.894 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-05-06T13:40:41.894 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=uname 2024-05-06T13:40:41.894 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=vmstat 2024-05-06T13:40:41.894 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=xfs 2024-05-06T13:40:41.894 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:115 level=info collector=zfs 2024-05-06T13:40:41.894 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.806Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-05-06T13:40:41.894 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[56376]: ts=2024-05-06T13:40:41.807Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-05-06T13:40:41.895 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 bash[56361]: 54aee5b22b1c8a81efb1e966694a1742bda9d691f9c9088d9f3dfc6b3907e93b 2024-05-06T13:40:42.302 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:40:41 smithi077 systemd[1]: Started Ceph node-exporter.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:40:42.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:42 smithi077 ceph-mon[22428]: Reconfiguring node-exporter.a (dependencies changed)... 2024-05-06T13:40:42.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:42 smithi077 ceph-mon[22428]: Reconfiguring daemon node-exporter.a on smithi077 2024-05-06T13:40:42.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:42 smithi077 ceph-mon[22428]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-06T13:40:42.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:42 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:42.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:42 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:42.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:42 smithi077 ceph-mon[27166]: Reconfiguring node-exporter.a (dependencies changed)... 2024-05-06T13:40:42.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:42 smithi077 ceph-mon[27166]: Reconfiguring daemon node-exporter.a on smithi077 2024-05-06T13:40:42.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:42 smithi077 ceph-mon[27166]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-06T13:40:42.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:42 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:42.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:42 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:42.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:42 smithi170 ceph-mon[23272]: Reconfiguring node-exporter.a (dependencies changed)... 2024-05-06T13:40:42.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:42 smithi170 ceph-mon[23272]: Reconfiguring daemon node-exporter.a on smithi077 2024-05-06T13:40:42.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:42 smithi170 ceph-mon[23272]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-06T13:40:42.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:42 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:42.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:42 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:43.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:43 smithi077 ceph-mon[22428]: Reconfiguring alertmanager.a (dependencies changed)... 2024-05-06T13:40:43.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:43 smithi077 ceph-mon[22428]: Reconfiguring daemon alertmanager.a on smithi077 2024-05-06T13:40:43.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:43 smithi077 ceph-mon[27166]: Reconfiguring alertmanager.a (dependencies changed)... 2024-05-06T13:40:43.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:43 smithi077 ceph-mon[27166]: Reconfiguring daemon alertmanager.a on smithi077 2024-05-06T13:40:43.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:43 smithi170 ceph-mon[23272]: Reconfiguring alertmanager.a (dependencies changed)... 2024-05-06T13:40:43.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:43 smithi170 ceph-mon[23272]: Reconfiguring daemon alertmanager.a on smithi077 2024-05-06T13:40:43.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=error ts=2024-05-06T13:40:43.517Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs" 2024-05-06T13:40:43.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:40:43.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.77:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.77 because it doesn't contain any IP SANs" 2024-05-06T13:40:43.802 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=warn ts=2024-05-06T13:40:43.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.170:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.170 because it doesn't contain any IP SANs" 2024-05-06T13:40:44.185 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:44 smithi077 ceph-mon[22428]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:40:44.185 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:44 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:40:44.185 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:44 smithi077 ceph-mon[27166]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:40:44.185 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:44 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:40:44.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:44 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:40:44] "GET /metrics HTTP/1.1" 200 37894 "" "Prometheus/2.33.4" 2024-05-06T13:40:44.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:44 smithi170 ceph-mon[23272]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:40:44.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:44 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:40:44.802 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:40:44 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[22660]: ::ffff:172.21.15.170 - - [06/May/2024:13:40:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:40:45.446 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:45 smithi077 systemd[1]: Stopping Ceph alertmanager.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:40:45.446 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:45 smithi077 bash[56553]: Error: no container with name or ID "ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager.a" found: no such container 2024-05-06T13:40:45.789 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:45 smithi077 ceph-mon[22428]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:40:45.790 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:45 smithi077 ceph-mon[27166]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:40:45.790 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:45 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[47253]: level=info ts=2024-05-06T13:40:45.446Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-05-06T13:40:45.790 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:45 smithi077 podman[56566]: 2024-05-06 13:40:45.466925247 +0000 UTC m=+0.144952669 container died d179964192632edb46dd4058afcc7a5e82f150f1a88cf8c0272d026b1442e1d9 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T13:40:46.052 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:45 smithi077 podman[56566]: 2024-05-06 13:40:45.880182733 +0000 UTC m=+0.558210114 container cleanup d179964192632edb46dd4058afcc7a5e82f150f1a88cf8c0272d026b1442e1d9 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T13:40:46.052 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:45 smithi077 bash[56566]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a 2024-05-06T13:40:46.052 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:45 smithi077 bash[56594]: Error: no container with name or ID "ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager.a" found: no such container 2024-05-06T13:40:46.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:45 smithi170 ceph-mon[23272]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:40:46.423 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:46 smithi077 podman[56581]: 2024-05-06 13:40:46.089555726 +0000 UTC m=+0.617159852 container remove d179964192632edb46dd4058afcc7a5e82f150f1a88cf8c0272d026b1442e1d9 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T13:40:46.423 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:46 smithi077 podman[56581]: 2024-05-06 13:40:46.190553904 +0000 UTC m=+0.718158033 volume remove 1b88354a20e32ecfa3a835d12b718d3f7fa5a437daa18ca512482b5161d6815f 2024-05-06T13:40:46.423 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:46 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-05-06T13:40:46.423 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:46 smithi077 systemd[1]: Stopped Ceph alertmanager.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:40:46.423 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:46 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@alertmanager.a.service: Consumed 1.051s CPU time. 2024-05-06T13:40:46.423 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:46 smithi077 systemd[1]: Starting Ceph alertmanager.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:40:46.682 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:46 smithi077 podman[56662]: 2024-05-06 13:40:46.326160301 +0000 UTC m=+0.018492401 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-05-06T13:40:46.682 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:46 smithi077 podman[56662]: 2024-05-06 13:40:46.441336771 +0000 UTC m=+0.133668878 volume create b6479ecfa0b406387bf58df1a23204dd25c00c04a98d58109ab829983e3507a4 2024-05-06T13:40:46.683 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:46 smithi077 podman[56662]: 2024-05-06 13:40:46.508526714 +0000 UTC m=+0.200858856 container create 0f855548777f89ed44bceeee061f31f3b5a40bcfc1e91dd8df0d4cbbdb01940e (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T13:40:47.019 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:46 smithi077 podman[56662]: 2024-05-06 13:40:46.803642602 +0000 UTC m=+0.495974703 container init 0f855548777f89ed44bceeee061f31f3b5a40bcfc1e91dd8df0d4cbbdb01940e (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T13:40:47.019 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:46 smithi077 podman[56662]: 2024-05-06 13:40:46.806306554 +0000 UTC m=+0.498638662 container start 0f855548777f89ed44bceeee061f31f3b5a40bcfc1e91dd8df0d4cbbdb01940e (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T13:40:47.019 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:46 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[56677]: level=info ts=2024-05-06T13:40:46.816Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-05-06T13:40:47.019 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:46 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[56677]: level=info ts=2024-05-06T13:40:46.816Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-05-06T13:40:47.019 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:46 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[56677]: level=info ts=2024-05-06T13:40:46.817Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.77 port=9094 2024-05-06T13:40:47.019 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:46 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[56677]: level=info ts=2024-05-06T13:40:46.818Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-05-06T13:40:47.020 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:46 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[56677]: level=info ts=2024-05-06T13:40:46.843Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-05-06T13:40:47.020 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:46 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[56677]: level=info ts=2024-05-06T13:40:46.843Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-05-06T13:40:47.020 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:46 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[56677]: level=info ts=2024-05-06T13:40:46.845Z caller=main.go:518 msg=Listening address=:9093 2024-05-06T13:40:47.020 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:46 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[56677]: level=info ts=2024-05-06T13:40:46.845Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-05-06T13:40:47.020 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:46 smithi077 bash[56662]: 0f855548777f89ed44bceeee061f31f3b5a40bcfc1e91dd8df0d4cbbdb01940e 2024-05-06T13:40:47.020 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:46 smithi077 systemd[1]: Started Ceph alertmanager.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:40:48.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:47 smithi077 ceph-mon[27166]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:40:48.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:47 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:48.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:47 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:48.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:47 smithi077 ceph-mon[27166]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:40:48.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:47 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:40:48.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:47 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:40:48.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:47 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:40:48.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:47 smithi077 ceph-mon[27166]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:40:48.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:47 smithi077 ceph-mon[22428]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:40:48.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:47 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:48.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:47 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:48.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:47 smithi077 ceph-mon[22428]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:40:48.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:47 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:40:48.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:47 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:40:48.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:47 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:40:48.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:47 smithi077 ceph-mon[22428]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:40:48.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:47 smithi170 ceph-mon[23272]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:40:48.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:47 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:48.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:47 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:48.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:47 smithi170 ceph-mon[23272]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:40:48.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:47 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:40:48.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:47 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:40:48.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:47 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:40:48.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:47 smithi170 ceph-mon[23272]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:40:49.073 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:48 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[56677]: level=info ts=2024-05-06T13:40:48.818Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000049698s 2024-05-06T13:40:50.161 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:49 smithi077 ceph-mon[22428]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:40:50.162 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:49 smithi077 ceph-mon[27166]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:40:50.263 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:49 smithi170 ceph-mon[23272]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:40:50.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:40:50.262+0000 7f893079f700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:40:50.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Creating ceph-iscsi config... 2024-05-06T13:40:50.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:40:50.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:40:50.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:40:50.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:40:50.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:40:50.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Traceback (most recent call last): 2024-05-06T13:40:50.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:40:50.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return _run_code(code, main_globals, None, 2024-05-06T13:40:50.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:40:50.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: exec(code, run_globals) 2024-05-06T13:40:50.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:40:50.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:40:50.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:40:50.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:40:50.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:40:50.578 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:40:50.578 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:40:50.578 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:40:50.578 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:40:50.578 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Traceback (most recent call last): 2024-05-06T13:40:50.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:40:50.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:40:50.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:40:50.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:40:50.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:40:50.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:40:50.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:40:50.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return future.result(timeout) 2024-05-06T13:40:50.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:40:50.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return self.__get_result() 2024-05-06T13:40:50.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:40:50.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: raise self._exception 2024-05-06T13:40:50.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:40:50.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: use_current_daemon_image=reconfig, 2024-05-06T13:40:50.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:40:50.580 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:40:50.580 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:40:50.580 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Creating ceph-iscsi config... 2024-05-06T13:40:50.580 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:40:50.580 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:40:50.580 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:40:50.580 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:40:50.580 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:40:50.580 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Traceback (most recent call last): 2024-05-06T13:40:50.580 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:40:50.581 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return _run_code(code, main_globals, None, 2024-05-06T13:40:50.581 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:40:50.581 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: exec(code, run_globals) 2024-05-06T13:40:50.581 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:40:50.581 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:40:50.581 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:40:50.581 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:40:50.581 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:40:50.581 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:40:50.581 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:40:50.581 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:40:50.581 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:40:52.292 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:51 smithi170 ceph-mon[23272]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:40:52.292 INFO:journalctl@ceph.mon.b.smithi170.stdout: Creating ceph-iscsi config... 2024-05-06T13:40:52.292 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:40:52.292 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:40:52.292 INFO:journalctl@ceph.mon.b.smithi170.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:40:52.292 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:40:52.292 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:40:52.292 INFO:journalctl@ceph.mon.b.smithi170.stdout: Traceback (most recent call last): 2024-05-06T13:40:52.292 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:40:52.293 INFO:journalctl@ceph.mon.b.smithi170.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:40:52.293 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:40:52.293 INFO:journalctl@ceph.mon.b.smithi170.stdout: exec(code, run_globals) 2024-05-06T13:40:52.293 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:40:52.293 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:40:52.293 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:40:52.293 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:40:52.293 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:40:52.293 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:40:52.293 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:40:52.293 INFO:journalctl@ceph.mon.b.smithi170.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:40:52.294 INFO:journalctl@ceph.mon.b.smithi170.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:40:52.294 INFO:journalctl@ceph.mon.b.smithi170.stdout: Traceback (most recent call last): 2024-05-06T13:40:52.294 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:40:52.294 INFO:journalctl@ceph.mon.b.smithi170.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:40:52.294 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:40:52.294 INFO:journalctl@ceph.mon.b.smithi170.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:40:52.294 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:40:52.294 INFO:journalctl@ceph.mon.b.smithi170.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:40:52.294 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:40:52.294 INFO:journalctl@ceph.mon.b.smithi170.stdout: return future.result(timeout) 2024-05-06T13:40:52.294 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:40:52.294 INFO:journalctl@ceph.mon.b.smithi170.stdout: return self.__get_result() 2024-05-06T13:40:52.294 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:40:52.295 INFO:journalctl@ceph.mon.b.smithi170.stdout: raise self._exception 2024-05-06T13:40:52.295 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:40:52.295 INFO:journalctl@ceph.mon.b.smithi170.stdout: use_current_daemon_image=reconfig, 2024-05-06T13:40:52.295 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:40:52.295 INFO:journalctl@ceph.mon.b.smithi170.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:40:52.295 INFO:journalctl@ceph.mon.b.smithi170.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:40:52.295 INFO:journalctl@ceph.mon.b.smithi170.stdout: Creating ceph-iscsi config... 2024-05-06T13:40:52.295 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:40:52.295 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:40:52.295 INFO:journalctl@ceph.mon.b.smithi170.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:40:52.295 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:40:52.295 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:40:52.295 INFO:journalctl@ceph.mon.b.smithi170.stdout: Traceback (most recent call last): 2024-05-06T13:40:52.295 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:40:52.295 INFO:journalctl@ceph.mon.b.smithi170.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:40:52.296 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:40:52.296 INFO:journalctl@ceph.mon.b.smithi170.stdout: exec(code, run_globals) 2024-05-06T13:40:52.296 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:40:52.296 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:40:52.296 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:40:52.296 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:40:52.296 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:40:52.296 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:40:52.296 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:40:52.296 INFO:journalctl@ceph.mon.b.smithi170.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:40:52.296 INFO:journalctl@ceph.mon.b.smithi170.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:40:52.296 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:51 smithi170 ceph-mon[23272]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-06T13:40:52.296 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:51 smithi170 ceph-mon[23272]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:40:52.296 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:51 smithi170 ceph-mon[23272]: Reconfiguring daemon prometheus.a on smithi170 2024-05-06T13:40:52.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:51 smithi077 ceph-mon[22428]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:40:52.302 INFO:journalctl@ceph.mon.a.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:40:52.302 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:40:52.302 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:40:52.302 INFO:journalctl@ceph.mon.a.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:40:52.303 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:40:52.303 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:40:52.303 INFO:journalctl@ceph.mon.a.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:40:52.303 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:40:52.303 INFO:journalctl@ceph.mon.a.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:40:52.303 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:40:52.303 INFO:journalctl@ceph.mon.a.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:40:52.303 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:40:52.303 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:40:52.303 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:40:52.303 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:40:52.303 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:40:52.303 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:40:52.304 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:40:52.304 INFO:journalctl@ceph.mon.a.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:40:52.304 INFO:journalctl@ceph.mon.a.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:40:52.304 INFO:journalctl@ceph.mon.a.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:40:52.304 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:40:52.304 INFO:journalctl@ceph.mon.a.smithi077.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:40:52.304 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:40:52.304 INFO:journalctl@ceph.mon.a.smithi077.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:40:52.304 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:40:52.304 INFO:journalctl@ceph.mon.a.smithi077.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:40:52.304 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:40:52.304 INFO:journalctl@ceph.mon.a.smithi077.stdout: return future.result(timeout) 2024-05-06T13:40:52.304 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:40:52.305 INFO:journalctl@ceph.mon.a.smithi077.stdout: return self.__get_result() 2024-05-06T13:40:52.305 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:40:52.305 INFO:journalctl@ceph.mon.a.smithi077.stdout: raise self._exception 2024-05-06T13:40:52.305 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:40:52.305 INFO:journalctl@ceph.mon.a.smithi077.stdout: use_current_daemon_image=reconfig, 2024-05-06T13:40:52.305 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:40:52.305 INFO:journalctl@ceph.mon.a.smithi077.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:40:52.305 INFO:journalctl@ceph.mon.a.smithi077.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:40:52.305 INFO:journalctl@ceph.mon.a.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:40:52.305 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:40:52.305 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:40:52.305 INFO:journalctl@ceph.mon.a.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:40:52.305 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:40:52.305 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:40:52.306 INFO:journalctl@ceph.mon.a.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:40:52.306 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:40:52.306 INFO:journalctl@ceph.mon.a.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:40:52.306 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:40:52.306 INFO:journalctl@ceph.mon.a.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:40:52.306 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:40:52.306 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:40:52.306 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:40:52.306 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:40:52.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:51 smithi077 ceph-mon[27166]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:40:52.307 INFO:journalctl@ceph.mon.c.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:40:52.307 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:40:52.307 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:40:52.307 INFO:journalctl@ceph.mon.c.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:40:52.307 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:40:52.307 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:40:52.307 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:40:52.307 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:40:52.307 INFO:journalctl@ceph.mon.c.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:40:52.307 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:40:52.307 INFO:journalctl@ceph.mon.c.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:40:52.307 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:40:52.307 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:40:52.307 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:40:52.308 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:40:52.308 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:40:52.308 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:40:52.308 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:40:52.308 INFO:journalctl@ceph.mon.c.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:40:52.308 INFO:journalctl@ceph.mon.c.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:40:52.308 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:40:52.308 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:40:52.308 INFO:journalctl@ceph.mon.c.smithi077.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:40:52.308 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:40:52.308 INFO:journalctl@ceph.mon.c.smithi077.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:40:52.308 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:40:52.308 INFO:journalctl@ceph.mon.c.smithi077.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:40:52.309 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:40:52.309 INFO:journalctl@ceph.mon.c.smithi077.stdout: return future.result(timeout) 2024-05-06T13:40:52.309 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:40:52.309 INFO:journalctl@ceph.mon.c.smithi077.stdout: return self.__get_result() 2024-05-06T13:40:52.309 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:40:52.309 INFO:journalctl@ceph.mon.c.smithi077.stdout: raise self._exception 2024-05-06T13:40:52.309 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:40:52.309 INFO:journalctl@ceph.mon.c.smithi077.stdout: use_current_daemon_image=reconfig, 2024-05-06T13:40:52.309 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:40:52.309 INFO:journalctl@ceph.mon.c.smithi077.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:40:52.309 INFO:journalctl@ceph.mon.c.smithi077.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:40:52.309 INFO:journalctl@ceph.mon.c.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:40:52.309 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:40:52.309 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:40:52.309 INFO:journalctl@ceph.mon.c.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:40:52.310 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:40:52.310 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:40:52.310 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:40:52.310 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:40:52.310 INFO:journalctl@ceph.mon.c.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:40:52.310 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:40:52.310 INFO:journalctl@ceph.mon.c.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:40:52.310 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:40:52.310 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:40:52.310 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:40:52.310 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:40:52.311 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:40:52.311 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:40:52.311 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:40:52.311 INFO:journalctl@ceph.mon.a.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:40:52.311 INFO:journalctl@ceph.mon.a.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:40:52.311 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:51 smithi077 ceph-mon[22428]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-06T13:40:52.312 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:51 smithi077 ceph-mon[22428]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:40:52.312 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:51 smithi077 ceph-mon[22428]: Reconfiguring daemon prometheus.a on smithi170 2024-05-06T13:40:52.312 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:40:52.312 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:40:52.312 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:40:52.312 INFO:journalctl@ceph.mon.c.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:40:52.312 INFO:journalctl@ceph.mon.c.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:40:52.312 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:51 smithi077 ceph-mon[27166]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-06T13:40:52.312 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:51 smithi077 ceph-mon[27166]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:40:52.312 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:51 smithi077 ceph-mon[27166]: Reconfiguring daemon prometheus.a on smithi170 2024-05-06T13:40:52.578 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:52 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:40:52.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.77:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.170:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:40:54.058 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:53 smithi170 ceph-mon[23272]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:40:54.058 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:53 smithi170 systemd[1]: Stopping Ceph prometheus.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:40:54.058 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:53 smithi170 bash[46522]: Error: no container with name or ID "ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus.a" found: no such container 2024-05-06T13:40:54.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:53 smithi077 ceph-mon[22428]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:40:54.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:53 smithi077 ceph-mon[27166]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:40:54.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:40:54 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:40:54] "GET /metrics HTTP/1.1" 200 37894 "" "Prometheus/2.33.4" 2024-05-06T13:40:54.326 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:54 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:40:54.091Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-05-06T13:40:54.326 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:54 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:40:54.091Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-05-06T13:40:54.326 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:54 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:40:54.091Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-05-06T13:40:54.326 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:54 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:40:54.091Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-05-06T13:40:54.326 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:54 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:40:54.091Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-05-06T13:40:54.326 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:54 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:40:54.091Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-05-06T13:40:54.327 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:54 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:40:54.092Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-05-06T13:40:54.327 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:54 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:40:54.092Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-05-06T13:40:54.327 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:54 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:40:54.092Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-05-06T13:40:54.327 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:54 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:40:54.092Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-05-06T13:40:54.327 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:54 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:40:54.092Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-05-06T13:40:54.327 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:54 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[40213]: ts=2024-05-06T13:40:54.092Z caller=main.go:1066 level=info msg="See you next time!" 2024-05-06T13:40:54.327 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:54 smithi170 podman[46535]: 2024-05-06 13:40:54.102325681 +0000 UTC m=+0.122197847 container died 488818ee0077edf750d1560079047234d8de5db5a9c23e04ba983c39f46b85dc (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:40:54.647 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:54 smithi170 podman[46535]: 2024-05-06 13:40:54.446226542 +0000 UTC m=+0.466098707 container cleanup 488818ee0077edf750d1560079047234d8de5db5a9c23e04ba983c39f46b85dc (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:40:54.647 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:54 smithi170 bash[46535]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a 2024-05-06T13:40:54.938 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:54 smithi170 podman[46549]: 2024-05-06 13:40:54.647592617 +0000 UTC m=+0.543089554 container remove 488818ee0077edf750d1560079047234d8de5db5a9c23e04ba983c39f46b85dc (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:40:54.938 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:54 smithi170 bash[46561]: Error: no container with name or ID "ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus.a" found: no such container 2024-05-06T13:40:54.938 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:54 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-05-06T13:40:54.938 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:54 smithi170 systemd[1]: Stopped Ceph prometheus.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:40:54.938 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:54 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@prometheus.a.service: Consumed 1.974s CPU time. 2024-05-06T13:40:54.938 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:54 smithi170 systemd[1]: Starting Ceph prometheus.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:40:55.200 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:54 smithi170 podman[46635]: 2024-05-06 13:40:54.842449388 +0000 UTC m=+0.019621189 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-05-06T13:40:55.200 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:54 smithi170 podman[46635]: 2024-05-06 13:40:54.959875434 +0000 UTC m=+0.137047278 container create c45c7b7e4059b598cd2b8032fd93d947a2d03a4c4e479b0ea38bd6ecf2a9fec8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:40:55.201 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:55 smithi170 podman[46635]: 2024-05-06 13:40:55.201147002 +0000 UTC m=+0.378318804 container init c45c7b7e4059b598cd2b8032fd93d947a2d03a4c4e479b0ea38bd6ecf2a9fec8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:40:55.577 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:55 smithi170 podman[46635]: 2024-05-06 13:40:55.204407731 +0000 UTC m=+0.381579533 container start c45c7b7e4059b598cd2b8032fd93d947a2d03a4c4e479b0ea38bd6ecf2a9fec8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:40:55.577 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:40:55.227Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-05-06T13:40:55.577 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:40:55.227Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-05-06T13:40:55.577 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:40:55.227Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-05-06T13:40:55.577 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:40:55.227Z caller=main.go:518 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi170 (none))" 2024-05-06T13:40:55.577 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:40:55.227Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-05-06T13:40:55.577 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:40:55.227Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-05-06T13:40:55.577 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:40:55.228Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-05-06T13:40:55.577 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:40:55.229Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-05-06T13:40:55.578 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:40:55.230Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-05-06T13:40:55.578 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:40:55.232Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-05-06T13:40:55.578 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:40:55.232Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.192µs 2024-05-06T13:40:55.578 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:40:55.232Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-05-06T13:40:55.578 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:55 smithi170 bash[46635]: c45c7b7e4059b598cd2b8032fd93d947a2d03a4c4e479b0ea38bd6ecf2a9fec8 2024-05-06T13:40:55.578 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:55 smithi170 systemd[1]: Started Ceph prometheus.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:40:55.578 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:40:55.446Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-05-06T13:40:55.954 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:55 smithi170 systemd[1]: Stopping Ceph node-exporter.b for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:40:55.954 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:55 smithi170 bash[46760]: Error: no container with name or ID "ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter.b" found: no such container 2024-05-06T13:40:55.955 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:40:55.666Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-05-06T13:40:55.955 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:40:55.666Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-05-06T13:40:55.955 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:40:55.666Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=40.994µs wal_replay_duration=433.504233ms total_replay_duration=433.559483ms 2024-05-06T13:40:55.955 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:40:55.669Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-05-06T13:40:55.955 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:40:55.669Z caller=main.go:947 level=info msg="TSDB started" 2024-05-06T13:40:55.955 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:40:55.669Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-05-06T13:40:55.955 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:40:55.691Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=21.674729ms db_storage=679ns remote_storage=1.43µs web_handler=301ns query_engine=711ns scrape=856.845µs scrape_sd=136.053µs notify=10.499µs notify_sd=8.998µs rules=20.427093ms 2024-05-06T13:40:55.955 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:40:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:40:55.691Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-05-06T13:40:56.254 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:55 smithi170 ceph-mon[23272]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:40:56.255 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:55 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:56.255 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:55 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:56.255 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:56 smithi170 podman[46773]: 2024-05-06 13:40:56.023012854 +0000 UTC m=+0.105598558 container died 001e61e5f9228963396904bf202c8180726e112d99ff8ab1167901230630367d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T13:40:56.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:55 smithi077 ceph-mon[22428]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:40:56.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:55 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:56.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:55 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:56.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:55 smithi077 ceph-mon[27166]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:40:56.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:55 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:56.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:55 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:56.576 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:56 smithi170 podman[46773]: 2024-05-06 13:40:56.421415794 +0000 UTC m=+0.504001488 container cleanup 001e61e5f9228963396904bf202c8180726e112d99ff8ab1167901230630367d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T13:40:56.576 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:56 smithi170 bash[46773]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b 2024-05-06T13:40:56.576 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:56 smithi170 bash[46800]: Error: no container with name or ID "ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter.b" found: no such container 2024-05-06T13:40:56.923 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:56 smithi170 podman[46787]: 2024-05-06 13:40:56.681121496 +0000 UTC m=+0.662291873 container remove 001e61e5f9228963396904bf202c8180726e112d99ff8ab1167901230630367d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T13:40:56.923 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:56 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-05-06T13:40:56.924 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:56 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-05-06T13:40:56.924 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:56 smithi170 systemd[1]: Stopped Ceph node-exporter.b for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:40:56.924 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:56 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@node-exporter.b.service: Consumed 3.851s CPU time. 2024-05-06T13:40:56.924 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:56 smithi170 systemd[1]: Starting Ceph node-exporter.b for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:40:57.233 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:56 smithi170 ceph-mon[23272]: Reconfiguring node-exporter.b (dependencies changed)... 2024-05-06T13:40:57.234 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:56 smithi170 ceph-mon[23272]: Reconfiguring daemon node-exporter.b on smithi170 2024-05-06T13:40:57.234 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:56 smithi170 podman[46881]: 2024-05-06 13:40:56.923783617 +0000 UTC m=+0.122527989 container create ca5df43a746919ea7a70320f840927261e67e18cc1f650ae5a508281b9abe044 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T13:40:57.234 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:56 smithi170 podman[46881]: 2024-05-06 13:40:56.826478255 +0000 UTC m=+0.025222626 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-05-06T13:40:57.234 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 podman[46881]: 2024-05-06 13:40:57.134663603 +0000 UTC m=+0.333407975 container init ca5df43a746919ea7a70320f840927261e67e18cc1f650ae5a508281b9abe044 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T13:40:57.234 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 podman[46881]: 2024-05-06 13:40:57.13732822 +0000 UTC m=+0.336072595 container start ca5df43a746919ea7a70320f840927261e67e18cc1f650ae5a508281b9abe044 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T13:40:57.235 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.140Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-05-06T13:40:57.235 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.140Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-05-06T13:40:57.235 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-05-06T13:40:57.235 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-05-06T13:40:57.235 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-05-06T13:40:57.235 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=arp 2024-05-06T13:40:57.235 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=bcache 2024-05-06T13:40:57.235 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=bonding 2024-05-06T13:40:57.235 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=btrfs 2024-05-06T13:40:57.235 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=conntrack 2024-05-06T13:40:57.235 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=cpu 2024-05-06T13:40:57.235 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-05-06T13:40:57.236 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=diskstats 2024-05-06T13:40:57.236 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=dmi 2024-05-06T13:40:57.236 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=edac 2024-05-06T13:40:57.236 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=entropy 2024-05-06T13:40:57.236 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-05-06T13:40:57.236 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=filefd 2024-05-06T13:40:57.236 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=filesystem 2024-05-06T13:40:57.236 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=hwmon 2024-05-06T13:40:57.236 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=infiniband 2024-05-06T13:40:57.236 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=ipvs 2024-05-06T13:40:57.236 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=loadavg 2024-05-06T13:40:57.237 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=mdadm 2024-05-06T13:40:57.237 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=meminfo 2024-05-06T13:40:57.237 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=netclass 2024-05-06T13:40:57.237 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=netdev 2024-05-06T13:40:57.237 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=netstat 2024-05-06T13:40:57.237 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=nfs 2024-05-06T13:40:57.237 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=nfsd 2024-05-06T13:40:57.237 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=nvme 2024-05-06T13:40:57.237 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=os 2024-05-06T13:40:57.237 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-05-06T13:40:57.237 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=pressure 2024-05-06T13:40:57.237 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=rapl 2024-05-06T13:40:57.237 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=schedstat 2024-05-06T13:40:57.237 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=sockstat 2024-05-06T13:40:57.237 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=softnet 2024-05-06T13:40:57.238 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=stat 2024-05-06T13:40:57.238 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=tapestats 2024-05-06T13:40:57.238 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=textfile 2024-05-06T13:40:57.238 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-05-06T13:40:57.238 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=time 2024-05-06T13:40:57.238 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-05-06T13:40:57.238 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=uname 2024-05-06T13:40:57.238 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=vmstat 2024-05-06T13:40:57.238 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=xfs 2024-05-06T13:40:57.238 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:115 level=info collector=zfs 2024-05-06T13:40:57.239 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-05-06T13:40:57.239 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[46897]: ts=2024-05-06T13:40:57.141Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-05-06T13:40:57.239 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 bash[46881]: ca5df43a746919ea7a70320f840927261e67e18cc1f650ae5a508281b9abe044 2024-05-06T13:40:57.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:56 smithi077 ceph-mon[22428]: Reconfiguring node-exporter.b (dependencies changed)... 2024-05-06T13:40:57.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:56 smithi077 ceph-mon[22428]: Reconfiguring daemon node-exporter.b on smithi170 2024-05-06T13:40:57.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:56 smithi077 ceph-mon[27166]: Reconfiguring node-exporter.b (dependencies changed)... 2024-05-06T13:40:57.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:56 smithi077 ceph-mon[27166]: Reconfiguring daemon node-exporter.b on smithi170 2024-05-06T13:40:57.303 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:40:56 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[56677]: level=info ts=2024-05-06T13:40:56.821Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002912569s 2024-05-06T13:40:57.576 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:40:57 smithi170 systemd[1]: Started Ceph node-exporter.b for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:40:58.272 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-mon[23272]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:40:58.272 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:58.272 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:58.272 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:58.272 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:58.273 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:57 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-06T13:40:58.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:57 smithi077 ceph-mon[22428]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:40:58.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:57 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:58.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:57 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:58.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:57 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:58.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:57 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:58.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:57 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-06T13:40:58.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:57 smithi077 ceph-mon[27166]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:40:58.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:57 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:58.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:57 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:58.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:57 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:58.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:57 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:40:58.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:57 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-06T13:40:59.287 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:58 smithi170 ceph-mon[23272]: Reconfiguring grafana.a (dependencies changed)... 2024-05-06T13:40:59.287 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:58 smithi170 ceph-mon[23272]: Regenerating cephadm self-signed grafana TLS certificates 2024-05-06T13:40:59.287 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:58 smithi170 ceph-mon[23272]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-06T13:40:59.287 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:58 smithi170 ceph-mon[23272]: Reconfiguring daemon grafana.a on smithi170 2024-05-06T13:40:59.287 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:58 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:40:59.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:58 smithi077 ceph-mon[27166]: Reconfiguring grafana.a (dependencies changed)... 2024-05-06T13:40:59.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:58 smithi077 ceph-mon[27166]: Regenerating cephadm self-signed grafana TLS certificates 2024-05-06T13:40:59.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:58 smithi077 ceph-mon[27166]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-06T13:40:59.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:58 smithi077 ceph-mon[27166]: Reconfiguring daemon grafana.a on smithi170 2024-05-06T13:40:59.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:58 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:40:59.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:58 smithi077 ceph-mon[22428]: Reconfiguring grafana.a (dependencies changed)... 2024-05-06T13:40:59.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:58 smithi077 ceph-mon[22428]: Regenerating cephadm self-signed grafana TLS certificates 2024-05-06T13:40:59.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:58 smithi077 ceph-mon[22428]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-06T13:40:59.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:58 smithi077 ceph-mon[22428]: Reconfiguring daemon grafana.a on smithi170 2024-05-06T13:40:59.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:58 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:41:00.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:40:59 smithi077 ceph-mon[22428]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-05-06T13:41:00.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:40:59 smithi077 ceph-mon[27166]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-05-06T13:41:00.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:40:59 smithi170 ceph-mon[23272]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-05-06T13:41:00.326 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:00 smithi170 systemd[1]: Stopping Ceph grafana.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:41:00.326 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:00 smithi170 bash[47072]: Error: no container with name or ID "ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana.a" found: no such container 2024-05-06T13:41:00.326 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[39414]: t=2024-05-06T13:41:00+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-05-06T13:41:00.326 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:00 smithi170 podman[47084]: 2024-05-06 13:41:00.204792995 +0000 UTC m=+0.135002163 container died 4b872c67236dfa9268c7b071d80c951285a2fb5b2b4be664c51f1e7016339621 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, com.redhat.component=ubi8-container, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, release=236.1648460182, description=Ceph Grafana Container, name=ubi8, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, vendor=Red Hat, Inc., io.buildah.version=1.24.2) 2024-05-06T13:41:00.826 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:00 smithi170 podman[47084]: 2024-05-06 13:41:00.628716844 +0000 UTC m=+0.558926015 container cleanup 4b872c67236dfa9268c7b071d80c951285a2fb5b2b4be664c51f1e7016339621 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, io.buildah.version=1.24.2, architecture=x86_64, description=Ceph Grafana Container, distribution-scope=public, version=8.5, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-05-06T13:41:00.826 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:00 smithi170 bash[47084]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a 2024-05-06T13:41:00.826 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:00 smithi170 bash[47110]: Error: no container with name or ID "ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana.a" found: no such container 2024-05-06T13:41:01.205 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:00 smithi170 podman[47098]: 2024-05-06 13:41:00.844025062 +0000 UTC m=+0.634344439 container remove 4b872c67236dfa9268c7b071d80c951285a2fb5b2b4be664c51f1e7016339621 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, version=8.5, description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, com.redhat.component=ubi8-container, distribution-scope=public, name=ubi8, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182) 2024-05-06T13:41:01.206 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:00 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@grafana.a.service: Deactivated successfully. 2024-05-06T13:41:01.206 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:00 smithi170 systemd[1]: Stopped Ceph grafana.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:41:01.206 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:00 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@grafana.a.service: Consumed 3.394s CPU time. 2024-05-06T13:41:01.206 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:00 smithi170 systemd[1]: Starting Ceph grafana.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:41:01.206 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 podman[47180]: 2024-05-06 13:41:00.973638259 +0000 UTC m=+0.017754633 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-05-06T13:41:01.206 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 podman[47180]: 2024-05-06 13:41:01.080757084 +0000 UTC m=+0.124873431 container create fdcecac7992557cf2777d9450df344219792984e400651bbc54db2739786f0b8 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a, release=236.1648460182, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.expose-services=, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, description=Ceph Grafana Container, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, build-date=2022-03-28T10:36:18.413762, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, distribution-scope=public) 2024-05-06T13:41:01.500 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 podman[47180]: 2024-05-06 13:41:01.339428987 +0000 UTC m=+0.383545346 container init fdcecac7992557cf2777d9450df344219792984e400651bbc54db2739786f0b8 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, maintainer=Paul Cuzner , vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, release=236.1648460182, name=ubi8) 2024-05-06T13:41:01.500 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 podman[47180]: 2024-05-06 13:41:01.342035939 +0000 UTC m=+0.386152292 container start fdcecac7992557cf2777d9450df344219792984e400651bbc54db2739786f0b8 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, version=8.5, architecture=x86_64, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, io.openshift.tags=base rhel8, build-date=2022-03-28T10:36:18.413762, release=236.1648460182, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-05-06T13:41:01.500 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-05-06T13:41:01.501 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-05-06T13:41:01.501 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-05-06T13:41:01.501 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-05-06T13:41:01.501 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-05-06T13:41:01.501 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-05-06T13:41:01.501 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-05-06T13:41:01.501 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-05-06T13:41:01.501 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-05-06T13:41:01.502 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-05-06T13:41:01.502 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-05-06T13:41:01.502 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-05-06T13:41:01.502 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-05-06T13:41:01.502 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="App mode production" logger=settings 2024-05-06T13:41:01.502 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-05-06T13:41:01.502 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-05-06T13:41:01.502 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-05-06T13:41:01.502 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=377 duration=548.093µs 2024-05-06T13:41:01.502 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-05-06T13:41:01.502 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-05-06T13:41:01.503 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-05-06T13:41:01.503 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 bash[47180]: fdcecac7992557cf2777d9450df344219792984e400651bbc54db2739786f0b8 2024-05-06T13:41:01.503 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 systemd[1]: Started Ceph grafana.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:41:01.503 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-05-06T13:41:01.503 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-05-06T13:41:01.503 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-05-06T13:41:01.503 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="deleted datasource based on configuration" logger=provisioning.datasources name=Dashboard1 2024-05-06T13:41:01.503 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-05-06T13:41:01.762 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Loki uid=P8E80F9AEF21F6940 2024-05-06T13:41:01.762 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-05-06T13:41:01.762 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-05-06T13:41:01.762 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:41:01+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-05-06T13:41:01.762 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:41:01] ENGINE Bus STOPPING 2024-05-06T13:41:01.762 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:41:01] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-06T13:41:01.762 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:41:01] ENGINE Bus STOPPED 2024-05-06T13:41:01.762 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:41:01] ENGINE Bus STARTING 2024-05-06T13:41:01.763 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:41:01] ENGINE Serving on http://:::9283 2024-05-06T13:41:01.763 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:41:01] ENGINE Bus STARTED 2024-05-06T13:41:01.763 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:41:01] ENGINE Bus STOPPING 2024-05-06T13:41:01.763 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:41:01] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-06T13:41:01.763 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:41:01] ENGINE Bus STOPPED 2024-05-06T13:41:01.763 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:41:01] ENGINE Bus STARTING 2024-05-06T13:41:01.763 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:41:01] ENGINE Serving on http://:::9283 2024-05-06T13:41:01.763 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:41:01] ENGINE Bus STARTED 2024-05-06T13:41:02.076 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:41:01] ENGINE Bus STOPPING 2024-05-06T13:41:02.076 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:41:01] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-06T13:41:02.076 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:41:01] ENGINE Bus STOPPED 2024-05-06T13:41:02.076 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:41:01] ENGINE Bus STARTING 2024-05-06T13:41:02.076 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:41:01] ENGINE Serving on http://:::9283 2024-05-06T13:41:02.076 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:41:01] ENGINE Bus STARTED 2024-05-06T13:41:02.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:01 smithi170 ceph-mon[23272]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-05-06T13:41:02.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:01 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:02.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:01 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:02.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:01 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T13:41:02.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:01 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi077.front.sepia.ceph.com:9093"}]: dispatch 2024-05-06T13:41:02.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:01 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:02.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:01 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:41:02.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:01 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://host.containers.internal:9095"}]: dispatch 2024-05-06T13:41:02.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:01 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:02.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:01 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T13:41:02.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:01 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://host.containers.internal:3000"}]: dispatch 2024-05-06T13:41:02.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:01 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:02.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:01 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2794808859' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:41:02.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:01 smithi077 ceph-mon[22428]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-05-06T13:41:02.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:01 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:02.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:01 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:02.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:01 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T13:41:02.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:01 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi077.front.sepia.ceph.com:9093"}]: dispatch 2024-05-06T13:41:02.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:01 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:02.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:01 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:41:02.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:01 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://host.containers.internal:9095"}]: dispatch 2024-05-06T13:41:02.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:01 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:02.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:01 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T13:41:02.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:01 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://host.containers.internal:3000"}]: dispatch 2024-05-06T13:41:02.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:01 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:02.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:01 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2794808859' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:41:02.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:01 smithi077 ceph-mon[27166]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-05-06T13:41:02.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:01 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:02.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:01 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:02.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:01 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T13:41:02.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:01 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi077.front.sepia.ceph.com:9093"}]: dispatch 2024-05-06T13:41:02.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:01 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:02.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:01 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:41:02.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:01 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://host.containers.internal:9095"}]: dispatch 2024-05-06T13:41:02.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:01 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:02.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:01 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T13:41:02.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:01 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://host.containers.internal:3000"}]: dispatch 2024-05-06T13:41:02.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:01 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:02.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:01 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2794808859' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:41:03.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:02 smithi077 ceph-mon[27166]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T13:41:03.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:02 smithi077 ceph-mon[27166]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi077.front.sepia.ceph.com:9093"}]: dispatch 2024-05-06T13:41:03.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:02 smithi077 ceph-mon[27166]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:41:03.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:02 smithi077 ceph-mon[27166]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://host.containers.internal:9095"}]: dispatch 2024-05-06T13:41:03.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:02 smithi077 ceph-mon[27166]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T13:41:03.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:02 smithi077 ceph-mon[27166]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://host.containers.internal:3000"}]: dispatch 2024-05-06T13:41:03.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:02 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2131173366' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2214497969"}]: dispatch 2024-05-06T13:41:03.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:02 smithi077 ceph-mon[22428]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T13:41:03.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:02 smithi077 ceph-mon[22428]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi077.front.sepia.ceph.com:9093"}]: dispatch 2024-05-06T13:41:03.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:02 smithi077 ceph-mon[22428]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:41:03.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:02 smithi077 ceph-mon[22428]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://host.containers.internal:9095"}]: dispatch 2024-05-06T13:41:03.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:02 smithi077 ceph-mon[22428]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T13:41:03.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:02 smithi077 ceph-mon[22428]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://host.containers.internal:3000"}]: dispatch 2024-05-06T13:41:03.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:02 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2131173366' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2214497969"}]: dispatch 2024-05-06T13:41:03.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:02 smithi170 ceph-mon[23272]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T13:41:03.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:02 smithi170 ceph-mon[23272]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi077.front.sepia.ceph.com:9093"}]: dispatch 2024-05-06T13:41:03.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:02 smithi170 ceph-mon[23272]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:41:03.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:02 smithi170 ceph-mon[23272]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://host.containers.internal:9095"}]: dispatch 2024-05-06T13:41:03.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:02 smithi170 ceph-mon[23272]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T13:41:03.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:02 smithi170 ceph-mon[23272]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://host.containers.internal:3000"}]: dispatch 2024-05-06T13:41:03.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:02 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2131173366' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2214497969"}]: dispatch 2024-05-06T13:41:04.194 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:03 smithi170 ceph-mon[23272]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:41:04.194 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:03 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2131173366' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2214497969"}]': finished 2024-05-06T13:41:04.194 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:03 smithi170 ceph-mon[23272]: osdmap e97: 8 total, 8 up, 8 in 2024-05-06T13:41:04.194 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:03 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/1815234681' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/930296124"}]: dispatch 2024-05-06T13:41:04.258 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:03 smithi077 ceph-mon[22428]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:41:04.258 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:03 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2131173366' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2214497969"}]': finished 2024-05-06T13:41:04.258 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:03 smithi077 ceph-mon[22428]: osdmap e97: 8 total, 8 up, 8 in 2024-05-06T13:41:04.258 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:03 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/1815234681' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/930296124"}]: dispatch 2024-05-06T13:41:04.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:03 smithi077 ceph-mon[27166]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:41:04.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:03 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2131173366' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2214497969"}]': finished 2024-05-06T13:41:04.259 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:03 smithi077 ceph-mon[27166]: osdmap e97: 8 total, 8 up, 8 in 2024-05-06T13:41:04.259 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:03 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/1815234681' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/930296124"}]: dispatch 2024-05-06T13:41:05.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:04 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/1815234681' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/930296124"}]': finished 2024-05-06T13:41:05.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:04 smithi077 ceph-mon[22428]: osdmap e98: 8 total, 8 up, 8 in 2024-05-06T13:41:05.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:04 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3525936103' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1241103129"}]: dispatch 2024-05-06T13:41:05.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:04 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/1815234681' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/930296124"}]': finished 2024-05-06T13:41:05.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:04 smithi077 ceph-mon[27166]: osdmap e98: 8 total, 8 up, 8 in 2024-05-06T13:41:05.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:04 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3525936103' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1241103129"}]: dispatch 2024-05-06T13:41:05.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:04 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/1815234681' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/930296124"}]': finished 2024-05-06T13:41:05.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:04 smithi170 ceph-mon[23272]: osdmap e98: 8 total, 8 up, 8 in 2024-05-06T13:41:05.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:04 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/3525936103' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1241103129"}]: dispatch 2024-05-06T13:41:06.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:05 smithi077 ceph-mon[22428]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:41:06.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:05 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3525936103' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1241103129"}]': finished 2024-05-06T13:41:06.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:05 smithi077 ceph-mon[22428]: osdmap e99: 8 total, 8 up, 8 in 2024-05-06T13:41:06.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:05 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:06.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:05 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:06.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:05 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2800860688' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2331082954"}]: dispatch 2024-05-06T13:41:06.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:05 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:06.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:05 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:06.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:05 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:41:06.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:05 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:41:06.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:05 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:06.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:05 smithi077 ceph-mon[27166]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:41:06.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:05 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3525936103' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1241103129"}]': finished 2024-05-06T13:41:06.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:05 smithi077 ceph-mon[27166]: osdmap e99: 8 total, 8 up, 8 in 2024-05-06T13:41:06.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:05 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:06.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:05 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:06.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:05 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2800860688' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2331082954"}]: dispatch 2024-05-06T13:41:06.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:05 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:06.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:05 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:06.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:05 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:41:06.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:05 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:41:06.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:05 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:06.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:05 smithi170 ceph-mon[23272]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:41:06.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:05 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/3525936103' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1241103129"}]': finished 2024-05-06T13:41:06.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:05 smithi170 ceph-mon[23272]: osdmap e99: 8 total, 8 up, 8 in 2024-05-06T13:41:06.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:05 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:06.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:05 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:06.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:05 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2800860688' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2331082954"}]: dispatch 2024-05-06T13:41:06.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:05 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:06.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:05 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:06.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:05 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:41:06.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:05 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:41:06.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:05 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:41:07.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:06 smithi077 ceph-mon[22428]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:41:07.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:06 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:41:07.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:06 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:41:07.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:06 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:41:07.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:06 smithi077 ceph-mon[22428]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:41:07.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:06 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2800860688' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2331082954"}]': finished 2024-05-06T13:41:07.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:06 smithi077 ceph-mon[22428]: osdmap e100: 8 total, 8 up, 8 in 2024-05-06T13:41:07.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:06 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/940999292' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/2284839445"}]: dispatch 2024-05-06T13:41:07.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:06 smithi077 ceph-mon[27166]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:41:07.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:06 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:41:07.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:06 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:41:07.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:06 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:41:07.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:06 smithi077 ceph-mon[27166]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:41:07.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:06 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2800860688' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2331082954"}]': finished 2024-05-06T13:41:07.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:06 smithi077 ceph-mon[27166]: osdmap e100: 8 total, 8 up, 8 in 2024-05-06T13:41:07.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:06 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/940999292' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/2284839445"}]: dispatch 2024-05-06T13:41:07.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:06 smithi170 ceph-mon[23272]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:41:07.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:06 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:41:07.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:06 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:41:07.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:06 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:41:07.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:06 smithi170 ceph-mon[23272]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:41:07.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:06 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2800860688' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2331082954"}]': finished 2024-05-06T13:41:07.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:06 smithi170 ceph-mon[23272]: osdmap e100: 8 total, 8 up, 8 in 2024-05-06T13:41:07.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:06 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/940999292' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/2284839445"}]: dispatch 2024-05-06T13:41:08.278 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:07 smithi077 ceph-mon[22428]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:41:08.278 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:07 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/940999292' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/2284839445"}]': finished 2024-05-06T13:41:08.278 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:07 smithi077 ceph-mon[22428]: osdmap e101: 8 total, 8 up, 8 in 2024-05-06T13:41:08.279 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:07 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/915615410' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/2284839445"}]: dispatch 2024-05-06T13:41:08.279 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:07 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/2284839445"}]: dispatch 2024-05-06T13:41:08.279 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:07 smithi077 ceph-mon[27166]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:41:08.279 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:07 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/940999292' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/2284839445"}]': finished 2024-05-06T13:41:08.279 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:07 smithi077 ceph-mon[27166]: osdmap e101: 8 total, 8 up, 8 in 2024-05-06T13:41:08.279 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:07 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/915615410' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/2284839445"}]: dispatch 2024-05-06T13:41:08.279 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:07 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/2284839445"}]: dispatch 2024-05-06T13:41:08.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:07 smithi170 ceph-mon[23272]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:41:08.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:07 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/940999292' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/2284839445"}]': finished 2024-05-06T13:41:08.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:07 smithi170 ceph-mon[23272]: osdmap e101: 8 total, 8 up, 8 in 2024-05-06T13:41:08.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:07 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/915615410' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/2284839445"}]: dispatch 2024-05-06T13:41:08.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:07 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/2284839445"}]: dispatch 2024-05-06T13:41:09.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:09 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/2284839445"}]': finished 2024-05-06T13:41:09.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:09 smithi077 ceph-mon[22428]: osdmap e102: 8 total, 8 up, 8 in 2024-05-06T13:41:09.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:09 smithi077 ceph-mon[22428]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:41:09.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:09 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/2284839445"}]': finished 2024-05-06T13:41:09.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:09 smithi077 ceph-mon[27166]: osdmap e102: 8 total, 8 up, 8 in 2024-05-06T13:41:09.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:09 smithi077 ceph-mon[27166]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:41:09.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:09 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/2284839445"}]': finished 2024-05-06T13:41:09.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:09 smithi170 ceph-mon[23272]: osdmap e102: 8 total, 8 up, 8 in 2024-05-06T13:41:09.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:09 smithi170 ceph-mon[23272]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:41:09.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:41:09] "GET /metrics HTTP/1.1" 200 37894 "" "Prometheus/2.33.4" 2024-05-06T13:41:09.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:41:09.345+0000 7f893079f700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:41:09.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Creating ceph-iscsi config... 2024-05-06T13:41:09.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:41:09.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:41:09.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:41:09.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:41:09.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:41:09.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Traceback (most recent call last): 2024-05-06T13:41:09.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:41:09.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return _run_code(code, main_globals, None, 2024-05-06T13:41:09.578 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:41:09.578 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: exec(code, run_globals) 2024-05-06T13:41:09.578 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:41:09.578 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:41:09.578 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:41:09.578 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:41:09.578 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:41:09.578 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:41:09.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:41:09.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:41:09.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:41:09.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Traceback (most recent call last): 2024-05-06T13:41:09.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:41:09.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:41:09.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:41:09.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:41:09.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:41:09.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:41:09.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:41:09.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return future.result(timeout) 2024-05-06T13:41:09.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:41:09.579 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return self.__get_result() 2024-05-06T13:41:09.580 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:41:09.580 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: raise self._exception 2024-05-06T13:41:09.580 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:41:09.580 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: use_current_daemon_image=reconfig, 2024-05-06T13:41:09.580 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:41:09.580 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:41:09.580 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:41:09.580 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Creating ceph-iscsi config... 2024-05-06T13:41:09.580 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:41:09.580 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:41:09.580 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:41:09.581 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:41:09.581 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:41:09.581 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Traceback (most recent call last): 2024-05-06T13:41:09.581 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:41:09.581 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return _run_code(code, main_globals, None, 2024-05-06T13:41:09.581 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:41:09.581 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: exec(code, run_globals) 2024-05-06T13:41:09.581 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:41:09.581 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:41:09.581 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:41:09.581 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:41:09.581 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:41:09.581 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:41:09.581 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:41:09.581 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:41:09.582 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:41:11.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:10 smithi077 ceph-mon[27166]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:41:11.053 INFO:journalctl@ceph.mon.c.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:41:11.053 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:41:11.053 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:41:11.053 INFO:journalctl@ceph.mon.c.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:41:11.053 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:41:11.053 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:41:11.053 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:41:11.053 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:41:11.053 INFO:journalctl@ceph.mon.c.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:41:11.053 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:41:11.053 INFO:journalctl@ceph.mon.c.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:41:11.053 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:41:11.054 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:41:11.054 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:41:11.054 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:41:11.054 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:41:11.054 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:41:11.054 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:41:11.054 INFO:journalctl@ceph.mon.c.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:41:11.054 INFO:journalctl@ceph.mon.c.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:41:11.054 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:41:11.054 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:41:11.054 INFO:journalctl@ceph.mon.c.smithi077.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:41:11.054 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:41:11.055 INFO:journalctl@ceph.mon.c.smithi077.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:41:11.055 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:41:11.055 INFO:journalctl@ceph.mon.c.smithi077.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:41:11.055 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:41:11.055 INFO:journalctl@ceph.mon.c.smithi077.stdout: return future.result(timeout) 2024-05-06T13:41:11.055 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:41:11.055 INFO:journalctl@ceph.mon.c.smithi077.stdout: return self.__get_result() 2024-05-06T13:41:11.055 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:41:11.055 INFO:journalctl@ceph.mon.c.smithi077.stdout: raise self._exception 2024-05-06T13:41:11.055 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:41:11.055 INFO:journalctl@ceph.mon.c.smithi077.stdout: use_current_daemon_image=reconfig, 2024-05-06T13:41:11.055 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:41:11.056 INFO:journalctl@ceph.mon.c.smithi077.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:41:11.056 INFO:journalctl@ceph.mon.c.smithi077.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:41:11.056 INFO:journalctl@ceph.mon.c.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:41:11.056 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:41:11.056 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:41:11.056 INFO:journalctl@ceph.mon.c.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:41:11.056 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:41:11.056 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:41:11.056 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:41:11.056 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:41:11.056 INFO:journalctl@ceph.mon.c.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:41:11.056 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:41:11.056 INFO:journalctl@ceph.mon.c.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:41:11.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:41:11.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:41:11.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:41:11.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:41:11.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:41:11.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:41:11.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:41:11.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:41:11.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:41:11.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:10 smithi077 ceph-mon[22428]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:41:11.058 INFO:journalctl@ceph.mon.a.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:41:11.058 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:41:11.058 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:41:11.058 INFO:journalctl@ceph.mon.a.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:41:11.058 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:41:11.058 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:41:11.058 INFO:journalctl@ceph.mon.a.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:41:11.058 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:41:11.058 INFO:journalctl@ceph.mon.a.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:41:11.058 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:41:11.058 INFO:journalctl@ceph.mon.a.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:41:11.059 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:41:11.059 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:41:11.059 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:41:11.059 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:41:11.059 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:41:11.059 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:41:11.059 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:41:11.059 INFO:journalctl@ceph.mon.a.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:41:11.059 INFO:journalctl@ceph.mon.a.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:41:11.059 INFO:journalctl@ceph.mon.a.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:41:11.059 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:41:11.059 INFO:journalctl@ceph.mon.a.smithi077.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:41:11.059 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:41:11.060 INFO:journalctl@ceph.mon.a.smithi077.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:41:11.060 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:41:11.060 INFO:journalctl@ceph.mon.a.smithi077.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:41:11.060 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:41:11.060 INFO:journalctl@ceph.mon.a.smithi077.stdout: return future.result(timeout) 2024-05-06T13:41:11.060 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:41:11.060 INFO:journalctl@ceph.mon.a.smithi077.stdout: return self.__get_result() 2024-05-06T13:41:11.060 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:41:11.060 INFO:journalctl@ceph.mon.a.smithi077.stdout: raise self._exception 2024-05-06T13:41:11.060 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:41:11.060 INFO:journalctl@ceph.mon.a.smithi077.stdout: use_current_daemon_image=reconfig, 2024-05-06T13:41:11.060 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:41:11.061 INFO:journalctl@ceph.mon.a.smithi077.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:41:11.061 INFO:journalctl@ceph.mon.a.smithi077.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:41:11.061 INFO:journalctl@ceph.mon.a.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:41:11.061 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:41:11.061 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:41:11.061 INFO:journalctl@ceph.mon.a.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:41:11.061 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:41:11.061 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:41:11.061 INFO:journalctl@ceph.mon.a.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:41:11.061 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:41:11.061 INFO:journalctl@ceph.mon.a.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:41:11.061 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:41:11.062 INFO:journalctl@ceph.mon.a.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:41:11.062 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:41:11.062 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:41:11.062 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:41:11.062 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:41:11.062 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:41:11.062 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:41:11.062 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:41:11.062 INFO:journalctl@ceph.mon.a.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:41:11.062 INFO:journalctl@ceph.mon.a.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:41:11.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:10 smithi170 ceph-mon[23272]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:41:11.076 INFO:journalctl@ceph.mon.b.smithi170.stdout: Creating ceph-iscsi config... 2024-05-06T13:41:11.076 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:41:11.076 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:41:11.076 INFO:journalctl@ceph.mon.b.smithi170.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:41:11.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:41:11.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:41:11.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: Traceback (most recent call last): 2024-05-06T13:41:11.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:41:11.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:41:11.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:41:11.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: exec(code, run_globals) 2024-05-06T13:41:11.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:41:11.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:41:11.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:41:11.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:41:11.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:41:11.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:41:11.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:41:11.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:41:11.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:41:11.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: Traceback (most recent call last): 2024-05-06T13:41:11.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:41:11.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:41:11.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:41:11.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:41:11.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:41:11.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:41:11.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:41:11.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: return future.result(timeout) 2024-05-06T13:41:11.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:41:11.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: return self.__get_result() 2024-05-06T13:41:11.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:41:11.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: raise self._exception 2024-05-06T13:41:11.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:41:11.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: use_current_daemon_image=reconfig, 2024-05-06T13:41:11.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:41:11.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:41:11.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:41:11.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: Creating ceph-iscsi config... 2024-05-06T13:41:11.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:41:11.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:41:11.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:41:11.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:41:11.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:41:11.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: Traceback (most recent call last): 2024-05-06T13:41:11.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:41:11.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:41:11.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:41:11.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: exec(code, run_globals) 2024-05-06T13:41:11.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:41:11.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:41:11.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:41:11.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:41:11.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:41:11.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:41:11.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:41:11.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:41:11.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:41:12.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:11 smithi077 ceph-mon[22428]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 1 op/s 2024-05-06T13:41:12.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:11 smithi077 ceph-mon[27166]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 1 op/s 2024-05-06T13:41:12.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:11 smithi170 ceph-mon[23272]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 1 op/s 2024-05-06T13:41:14.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:13 smithi077 ceph-mon[22428]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 847 B/s rd, 0 op/s 2024-05-06T13:41:14.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:13 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:41:14.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:13 smithi077 ceph-mon[27166]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 847 B/s rd, 0 op/s 2024-05-06T13:41:14.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:13 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:41:14.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:13 smithi170 ceph-mon[23272]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 847 B/s rd, 0 op/s 2024-05-06T13:41:14.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:13 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:41:16.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:15 smithi077 ceph-mon[27166]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-05-06T13:41:16.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:15 smithi077 ceph-mon[22428]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-05-06T13:41:16.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:15 smithi170 ceph-mon[23272]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-05-06T13:41:18.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:17 smithi077 ceph-mon[22428]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 567 B/s rd, 0 op/s 2024-05-06T13:41:18.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:17 smithi077 ceph-mon[27166]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 567 B/s rd, 0 op/s 2024-05-06T13:41:18.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:17 smithi170 ceph-mon[23272]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 567 B/s rd, 0 op/s 2024-05-06T13:41:19.076 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:41:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:41:18.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:41:19.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:19 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:41:19] "GET /metrics HTTP/1.1" 200 37893 "" "Prometheus/2.33.4" 2024-05-06T13:41:19.930 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:19 smithi077 ceph-mon[22428]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 510 B/s rd, 0 op/s 2024-05-06T13:41:19.930 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:19 smithi077 ceph-mon[27166]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 510 B/s rd, 0 op/s 2024-05-06T13:41:20.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:19 smithi170 ceph-mon[23272]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 510 B/s rd, 0 op/s 2024-05-06T13:41:22.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:21 smithi077 ceph-mon[22428]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-05-06T13:41:22.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:21 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/1326731197' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:41:22.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:21 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/4227055925' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1041195527"}]: dispatch 2024-05-06T13:41:22.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:21 smithi077 ceph-mon[27166]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-05-06T13:41:22.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:21 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/1326731197' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:41:22.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:21 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/4227055925' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1041195527"}]: dispatch 2024-05-06T13:41:22.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:21 smithi170 ceph-mon[23272]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-05-06T13:41:22.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:21 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/1326731197' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:41:22.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:21 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/4227055925' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1041195527"}]: dispatch 2024-05-06T13:41:23.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:22 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/4227055925' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1041195527"}]': finished 2024-05-06T13:41:23.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:22 smithi077 ceph-mon[27166]: osdmap e103: 8 total, 8 up, 8 in 2024-05-06T13:41:23.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:22 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/4227055925' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1041195527"}]': finished 2024-05-06T13:41:23.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:22 smithi077 ceph-mon[22428]: osdmap e103: 8 total, 8 up, 8 in 2024-05-06T13:41:23.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:22 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/4227055925' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1041195527"}]': finished 2024-05-06T13:41:23.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:22 smithi170 ceph-mon[23272]: osdmap e103: 8 total, 8 up, 8 in 2024-05-06T13:41:24.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:23 smithi077 ceph-mon[22428]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:41:24.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:23 smithi077 ceph-mon[27166]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:41:24.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:23 smithi170 ceph-mon[23272]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:41:26.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:25 smithi077 ceph-mon[22428]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 0 op/s 2024-05-06T13:41:26.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:25 smithi077 ceph-mon[27166]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 0 op/s 2024-05-06T13:41:26.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:25 smithi170 ceph-mon[23272]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 0 op/s 2024-05-06T13:41:28.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:27 smithi077 ceph-mon[22428]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 0 op/s 2024-05-06T13:41:28.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:27 smithi077 ceph-mon[27166]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 0 op/s 2024-05-06T13:41:28.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:27 smithi170 ceph-mon[23272]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 0 op/s 2024-05-06T13:41:29.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:28 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:41:29.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:28 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:41:29.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:28 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:41:29.076 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:41:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:41:28.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:41:29.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:41:29] "GET /metrics HTTP/1.1" 200 37884 "" "Prometheus/2.33.4" 2024-05-06T13:41:30.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:29 smithi077 ceph-mon[22428]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 0 op/s 2024-05-06T13:41:30.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:29 smithi077 ceph-mon[27166]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 0 op/s 2024-05-06T13:41:30.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:29 smithi170 ceph-mon[23272]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 0 op/s 2024-05-06T13:41:32.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:31 smithi077 ceph-mon[22428]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-05-06T13:41:32.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:31 smithi077 ceph-mon[22428]: from='client.25060 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:41:32.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:31 smithi077 ceph-mon[27166]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-05-06T13:41:32.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:31 smithi077 ceph-mon[27166]: from='client.25060 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:41:32.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:31 smithi170 ceph-mon[23272]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-05-06T13:41:32.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:31 smithi170 ceph-mon[23272]: from='client.25060 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:41:34.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:33 smithi077 ceph-mon[22428]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 858 B/s rd, 0 op/s 2024-05-06T13:41:34.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:33 smithi077 ceph-mon[27166]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 858 B/s rd, 0 op/s 2024-05-06T13:41:34.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:33 smithi170 ceph-mon[23272]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 858 B/s rd, 0 op/s 2024-05-06T13:41:36.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:35 smithi077 ceph-mon[22428]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:41:36.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:35 smithi077 ceph-mon[27166]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:41:36.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:35 smithi170 ceph-mon[23272]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:41:38.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:37 smithi077 ceph-mon[22428]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:41:38.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:37 smithi077 ceph-mon[27166]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:41:38.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:37 smithi170 ceph-mon[23272]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:41:39.076 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:41:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:41:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:41:39.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:39 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:41:39] "GET /metrics HTTP/1.1" 200 37884 "" "Prometheus/2.33.4" 2024-05-06T13:41:40.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:39 smithi077 ceph-mon[22428]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:41:40.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:39 smithi077 ceph-mon[27166]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:41:40.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:39 smithi170 ceph-mon[23272]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:41:42.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:41 smithi077 ceph-mon[22428]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:41:42.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:41 smithi077 ceph-mon[22428]: from='client.25060 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:41:42.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:41 smithi077 ceph-mon[27166]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:41:42.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:41 smithi077 ceph-mon[27166]: from='client.25060 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:41:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:41 smithi170 ceph-mon[23272]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:41:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:41 smithi170 ceph-mon[23272]: from='client.25060 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:41:44.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:43 smithi077 ceph-mon[22428]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:41:44.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:43 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:41:44.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:43 smithi077 ceph-mon[27166]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:41:44.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:43 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:41:44.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:43 smithi170 ceph-mon[23272]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:41:44.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:43 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:41:46.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:45 smithi077 ceph-mon[22428]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:41:46.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:45 smithi077 ceph-mon[27166]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:41:46.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:45 smithi170 ceph-mon[23272]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:41:48.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:47 smithi077 ceph-mon[22428]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:41:48.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:47 smithi077 ceph-mon[27166]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:41:48.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:47 smithi170 ceph-mon[23272]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:41:49.076 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:41:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:41:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:41:49.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:49 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:41:49] "GET /metrics HTTP/1.1" 200 37886 "" "Prometheus/2.33.4" 2024-05-06T13:41:50.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:49 smithi077 ceph-mon[22428]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:41:50.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:49 smithi077 ceph-mon[27166]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:41:50.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:49 smithi170 ceph-mon[23272]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:41:52.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:51 smithi077 ceph-mon[27166]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:41:52.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:51 smithi077 ceph-mon[27166]: from='client.25060 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:41:52.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:51 smithi077 ceph-mon[22428]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:41:52.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:51 smithi077 ceph-mon[22428]: from='client.25060 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:41:52.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:51 smithi170 ceph-mon[23272]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:41:52.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:51 smithi170 ceph-mon[23272]: from='client.25060 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:41:54.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:53 smithi077 ceph-mon[22428]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:41:54.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:53 smithi077 ceph-mon[27166]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:41:54.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:53 smithi170 ceph-mon[23272]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:41:56.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:55 smithi077 ceph-mon[22428]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:41:56.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:55 smithi077 ceph-mon[27166]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:41:56.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:55 smithi170 ceph-mon[23272]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:41:58.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:57 smithi077 ceph-mon[22428]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:41:58.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:57 smithi077 ceph-mon[27166]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:41:58.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:57 smithi170 ceph-mon[23272]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:41:59.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:58 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:41:59.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:58 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:41:59.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:58 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:41:59.077 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:41:58 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:41:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:41:59.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:41:59 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:41:59] "GET /metrics HTTP/1.1" 200 37888 "" "Prometheus/2.33.4" 2024-05-06T13:42:00.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:41:59 smithi077 ceph-mon[22428]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:00.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:41:59 smithi077 ceph-mon[27166]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:00.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:41:59 smithi170 ceph-mon[23272]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:02.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:01 smithi077 ceph-mon[22428]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:42:02.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:01 smithi077 ceph-mon[22428]: from='client.25060 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:42:02.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:01 smithi077 ceph-mon[27166]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:42:02.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:01 smithi077 ceph-mon[27166]: from='client.25060 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:42:02.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:01 smithi170 ceph-mon[23272]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:42:02.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:01 smithi170 ceph-mon[23272]: from='client.25060 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:42:04.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:03 smithi077 ceph-mon[22428]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:04.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:03 smithi077 ceph-mon[27166]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:04.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:03 smithi170 ceph-mon[23272]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:06.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:05 smithi077 ceph-mon[22428]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:42:06.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:05 smithi077 ceph-mon[27166]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:42:06.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:05 smithi170 ceph-mon[23272]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:42:08.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:07 smithi077 ceph-mon[22428]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:08.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:07 smithi077 ceph-mon[27166]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:08.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:07 smithi170 ceph-mon[23272]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:09.076 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:42:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:42:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:42:09.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:42:09] "GET /metrics HTTP/1.1" 200 37888 "" "Prometheus/2.33.4" 2024-05-06T13:42:10.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:09 smithi077 ceph-mon[22428]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:10.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:09 smithi077 ceph-mon[27166]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:10.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:09 smithi170 ceph-mon[23272]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:11.993 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:11 smithi170 ceph-mon[23272]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:42:11.993 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:11 smithi170 ceph-mon[23272]: from='client.25060 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:42:12.050 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:11 smithi077 ceph-mon[22428]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:42:12.050 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:11 smithi077 ceph-mon[22428]: from='client.25060 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:42:12.050 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:11 smithi077 ceph-mon[27166]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:42:12.050 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:11 smithi077 ceph-mon[27166]: from='client.25060 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:42:14.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:13 smithi077 ceph-mon[22428]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:14.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:13 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:42:14.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:13 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:42:14.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:13 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:42:14.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:13 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:42:14.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:13 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:42:14.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:13 smithi077 ceph-mon[27166]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:14.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:13 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:42:14.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:13 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:42:14.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:13 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:42:14.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:13 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:42:14.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:13 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:42:14.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:13 smithi170 ceph-mon[23272]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:14.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:13 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:42:14.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:13 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:42:14.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:13 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:42:14.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:13 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:42:14.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:13 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:42:15.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:14 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:42:15.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:14 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:42:15.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:14 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:42:15.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:14 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:42:15.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:14 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:42:15.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:14 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:42:15.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:14 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:42:15.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:14 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:42:15.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:14 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:42:15.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:14 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:42:15.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:14 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:42:15.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:14 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:42:15.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:14 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:42:15.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:14 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:42:15.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:14 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:42:15.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:14 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:42:15.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:14 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:42:15.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:14 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:42:16.013 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:15 smithi077 ceph-mon[22428]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:42:16.014 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:15 smithi077 ceph-mon[22428]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:42:16.014 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:15 smithi077 ceph-mon[22428]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:42:16.014 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:15 smithi077 ceph-mon[27166]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:42:16.014 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:15 smithi077 ceph-mon[27166]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:42:16.014 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:15 smithi077 ceph-mon[27166]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:42:16.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:15 smithi170 ceph-mon[23272]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:42:16.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:15 smithi170 ceph-mon[23272]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:42:16.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:15 smithi170 ceph-mon[23272]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:42:17.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:17 smithi170 ceph-mon[23272]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:17.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:42:17.528+0000 7f893079f700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:42:17.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Creating ceph-iscsi config... 2024-05-06T13:42:17.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:42:17.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:42:17.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:42:17.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:42:17.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:42:17.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Traceback (most recent call last): 2024-05-06T13:42:17.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:42:17.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return _run_code(code, main_globals, None, 2024-05-06T13:42:17.828 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:42:17.828 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: exec(code, run_globals) 2024-05-06T13:42:17.828 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:42:17.828 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:42:17.828 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:42:17.828 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:42:17.828 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:42:17.828 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:42:17.828 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:42:17.829 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:42:17.829 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:42:17.829 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Traceback (most recent call last): 2024-05-06T13:42:17.829 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:42:17.829 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:42:17.829 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:42:17.829 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:42:17.829 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:42:17.829 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:42:17.829 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:42:17.829 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return future.result(timeout) 2024-05-06T13:42:17.830 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:42:17.830 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return self.__get_result() 2024-05-06T13:42:17.830 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:42:17.830 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: raise self._exception 2024-05-06T13:42:17.831 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:42:17.831 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: use_current_daemon_image=reconfig, 2024-05-06T13:42:17.831 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:42:17.831 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:42:17.831 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:42:17.831 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Creating ceph-iscsi config... 2024-05-06T13:42:17.831 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:42:17.831 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:42:17.831 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:42:17.831 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:42:17.831 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:42:17.831 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Traceback (most recent call last): 2024-05-06T13:42:17.831 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:42:17.831 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return _run_code(code, main_globals, None, 2024-05-06T13:42:17.831 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:42:17.832 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: exec(code, run_globals) 2024-05-06T13:42:17.832 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:42:17.832 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:42:17.832 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:42:17.832 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:42:17.832 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:42:17.832 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:42:17.832 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:42:17.832 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:42:17.832 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:42:18.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:17 smithi077 ceph-mon[22428]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:18.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:17 smithi077 ceph-mon[27166]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:19.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:18 smithi077 ceph-mon[22428]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:42:19.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:42:19.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:42:19.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:42:19.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:42:19.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:42:19.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:42:19.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:42:19.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:42:19.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:42:19.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:42:19.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:42:19.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:42:19.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:42:19.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:42:19.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:42:19.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:42:19.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:42:19.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:42:19.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:42:19.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:42:19.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:42:19.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:42:19.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:42:19.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:42:19.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:42:19.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:42:19.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:42:19.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:42:19.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: return future.result(timeout) 2024-05-06T13:42:19.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:42:19.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: return self.__get_result() 2024-05-06T13:42:19.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:42:19.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: raise self._exception 2024-05-06T13:42:19.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:42:19.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: use_current_daemon_image=reconfig, 2024-05-06T13:42:19.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:42:19.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:42:19.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:42:19.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:42:19.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:42:19.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:42:19.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:42:19.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:42:19.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:42:19.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:42:19.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:42:19.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:42:19.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:42:19.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:42:19.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:42:19.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:42:19.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:42:19.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:42:19.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:42:19.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:42:19.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:42:19.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:42:19.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:42:19.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:18 smithi077 ceph-mon[27166]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:42:19.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:42:19.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:42:19.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:42:19.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:42:19.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:42:19.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:42:19.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:42:19.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:42:19.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:42:19.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:42:19.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:42:19.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:42:19.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:42:19.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:42:19.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:42:19.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:42:19.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:42:19.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:42:19.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:42:19.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:42:19.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:42:19.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:42:19.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:42:19.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:42:19.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:42:19.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:42:19.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:42:19.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:42:19.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: return future.result(timeout) 2024-05-06T13:42:19.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:42:19.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: return self.__get_result() 2024-05-06T13:42:19.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:42:19.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: raise self._exception 2024-05-06T13:42:19.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:42:19.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: use_current_daemon_image=reconfig, 2024-05-06T13:42:19.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:42:19.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:42:19.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:42:19.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:42:19.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:42:19.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:42:19.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:42:19.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:42:19.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:42:19.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:42:19.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:42:19.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:42:19.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:42:19.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:42:19.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:42:19.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:42:19.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:42:19.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:42:19.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:42:19.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:42:19.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:42:19.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:42:19.062 INFO:journalctl@ceph.mon.c.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:42:19.076 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:42:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:42:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:42:19.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:18 smithi170 ceph-mon[23272]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:42:19.076 INFO:journalctl@ceph.mon.b.smithi170.stdout: Creating ceph-iscsi config... 2024-05-06T13:42:19.076 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:42:19.076 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:42:19.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:42:19.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:42:19.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:42:19.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: Traceback (most recent call last): 2024-05-06T13:42:19.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:42:19.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:42:19.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:42:19.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: exec(code, run_globals) 2024-05-06T13:42:19.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:42:19.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:42:19.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:42:19.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:42:19.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:42:19.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:42:19.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:42:19.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:42:19.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:42:19.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: Traceback (most recent call last): 2024-05-06T13:42:19.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:42:19.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:42:19.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:42:19.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:42:19.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:42:19.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:42:19.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:42:19.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: return future.result(timeout) 2024-05-06T13:42:19.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:42:19.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: return self.__get_result() 2024-05-06T13:42:19.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:42:19.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: raise self._exception 2024-05-06T13:42:19.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:42:19.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: use_current_daemon_image=reconfig, 2024-05-06T13:42:19.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:42:19.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:42:19.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:42:19.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: Creating ceph-iscsi config... 2024-05-06T13:42:19.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:42:19.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:42:19.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:42:19.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:42:19.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:42:19.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: Traceback (most recent call last): 2024-05-06T13:42:19.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:42:19.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:42:19.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:42:19.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: exec(code, run_globals) 2024-05-06T13:42:19.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:42:19.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:42:19.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:42:19.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:42:19.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:42:19.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:42:19.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:42:19.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:42:19.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:42:19.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:19 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:42:19] "GET /metrics HTTP/1.1" 200 37888 "" "Prometheus/2.33.4" 2024-05-06T13:42:20.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:19 smithi077 ceph-mon[22428]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:20.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:19 smithi077 ceph-mon[27166]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:20.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:19 smithi170 ceph-mon[23272]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:22.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:21 smithi077 ceph-mon[22428]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T13:42:22.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:21 smithi077 ceph-mon[27166]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T13:42:22.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:21 smithi170 ceph-mon[23272]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T13:42:24.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:23 smithi077 ceph-mon[22428]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-05-06T13:42:24.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:23 smithi077 ceph-mon[27166]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-05-06T13:42:24.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:23 smithi170 ceph-mon[23272]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-05-06T13:42:26.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:25 smithi077 ceph-mon[22428]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-05-06T13:42:26.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:25 smithi077 ceph-mon[27166]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-05-06T13:42:26.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:25 smithi170 ceph-mon[23272]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-05-06T13:42:27.821 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:27 smithi077 ceph-mon[22428]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-05-06T13:42:27.821 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:27 smithi077 ceph-mon[27166]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-05-06T13:42:28.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:27 smithi170 ceph-mon[23272]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-05-06T13:42:29.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:28 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:42:29.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:28 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:42:29.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:28 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:42:29.076 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:42:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:42:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:42:29.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:42:29] "GET /metrics HTTP/1.1" 200 37882 "" "Prometheus/2.33.4" 2024-05-06T13:42:30.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:29 smithi077 ceph-mon[22428]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-05-06T13:42:30.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:29 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/4173040997' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:42:30.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:29 smithi077 ceph-mon[27166]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-05-06T13:42:30.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:29 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/4173040997' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:42:30.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:29 smithi170 ceph-mon[23272]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-05-06T13:42:30.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:29 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/4173040997' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:42:32.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:31 smithi077 ceph-mon[22428]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-05-06T13:42:32.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:31 smithi077 ceph-mon[27166]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-05-06T13:42:32.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:31 smithi170 ceph-mon[23272]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-05-06T13:42:34.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:33 smithi077 ceph-mon[22428]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-05-06T13:42:34.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:33 smithi077 ceph-mon[27166]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-05-06T13:42:34.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:33 smithi170 ceph-mon[23272]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-05-06T13:42:36.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:35 smithi077 ceph-mon[22428]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-06T13:42:36.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:35 smithi077 ceph-mon[27166]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-06T13:42:36.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:35 smithi170 ceph-mon[23272]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-06T13:42:38.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:37 smithi077 ceph-mon[22428]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-06T13:42:38.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:37 smithi077 ceph-mon[27166]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-06T13:42:38.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:37 smithi170 ceph-mon[23272]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-06T13:42:39.076 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:42:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:42:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:42:39.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:39 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:42:39] "GET /metrics HTTP/1.1" 200 37882 "" "Prometheus/2.33.4" 2024-05-06T13:42:40.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:39 smithi077 ceph-mon[22428]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-06T13:42:40.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:39 smithi077 ceph-mon[22428]: from='client.25090 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:42:40.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:39 smithi077 ceph-mon[27166]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-06T13:42:40.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:39 smithi077 ceph-mon[27166]: from='client.25090 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:42:40.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:39 smithi170 ceph-mon[23272]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-06T13:42:40.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:39 smithi170 ceph-mon[23272]: from='client.25090 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:42:42.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:41 smithi077 ceph-mon[22428]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-05-06T13:42:42.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:41 smithi077 ceph-mon[27166]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-05-06T13:42:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:41 smithi170 ceph-mon[23272]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-05-06T13:42:44.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:43 smithi077 ceph-mon[22428]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:44.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:43 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:42:44.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:43 smithi077 ceph-mon[27166]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:44.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:43 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:42:44.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:43 smithi170 ceph-mon[23272]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:44.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:43 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:42:46.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:45 smithi077 ceph-mon[22428]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:42:46.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:45 smithi077 ceph-mon[27166]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:42:46.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:45 smithi170 ceph-mon[23272]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:42:48.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:47 smithi077 ceph-mon[22428]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:48.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:47 smithi077 ceph-mon[27166]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:48.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:47 smithi170 ceph-mon[23272]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:49.076 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:42:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:42:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:42:49.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:49 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:42:49] "GET /metrics HTTP/1.1" 200 37893 "" "Prometheus/2.33.4" 2024-05-06T13:42:50.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:49 smithi077 ceph-mon[22428]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:50.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:49 smithi077 ceph-mon[22428]: from='client.25090 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:42:50.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:49 smithi077 ceph-mon[27166]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:50.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:49 smithi077 ceph-mon[27166]: from='client.25090 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:42:50.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:49 smithi170 ceph-mon[23272]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:50.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:49 smithi170 ceph-mon[23272]: from='client.25090 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:42:52.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:51 smithi077 ceph-mon[22428]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:42:52.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:51 smithi077 ceph-mon[27166]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:42:52.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:51 smithi170 ceph-mon[23272]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:42:54.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:53 smithi077 ceph-mon[22428]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:54.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:53 smithi077 ceph-mon[27166]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:54.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:53 smithi170 ceph-mon[23272]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:56.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:55 smithi077 ceph-mon[22428]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:42:56.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:55 smithi077 ceph-mon[27166]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:42:56.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:55 smithi170 ceph-mon[23272]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:42:58.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:57 smithi077 ceph-mon[22428]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:58.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:57 smithi077 ceph-mon[27166]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:58.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:57 smithi170 ceph-mon[23272]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:42:59.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:58 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:42:59.076 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:42:58 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:42:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:42:59.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:58 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:42:59.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:58 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:42:59.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:42:59 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:42:59] "GET /metrics HTTP/1.1" 200 37898 "" "Prometheus/2.33.4" 2024-05-06T13:43:00.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:59 smithi170 ceph-mon[23272]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:00.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:42:59 smithi170 ceph-mon[23272]: from='client.25090 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:43:00.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:59 smithi077 ceph-mon[22428]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:00.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:42:59 smithi077 ceph-mon[22428]: from='client.25090 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:43:00.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:59 smithi077 ceph-mon[27166]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:00.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:42:59 smithi077 ceph-mon[27166]: from='client.25090 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:43:02.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:01 smithi170 ceph-mon[23272]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:43:02.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:01 smithi077 ceph-mon[22428]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:43:02.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:01 smithi077 ceph-mon[27166]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:43:04.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:03 smithi170 ceph-mon[23272]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:04.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:03 smithi077 ceph-mon[22428]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:04.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:03 smithi077 ceph-mon[27166]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:06.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:05 smithi170 ceph-mon[23272]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:43:06.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:05 smithi077 ceph-mon[22428]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:43:06.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:05 smithi077 ceph-mon[27166]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:43:08.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:07 smithi170 ceph-mon[23272]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:08.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:07 smithi077 ceph-mon[22428]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:08.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:07 smithi077 ceph-mon[27166]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:09.076 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:43:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:43:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:43:09.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:43:09] "GET /metrics HTTP/1.1" 200 37898 "" "Prometheus/2.33.4" 2024-05-06T13:43:10.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:09 smithi077 ceph-mon[27166]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:10.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:09 smithi077 ceph-mon[27166]: from='client.25090 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:43:10.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:09 smithi077 ceph-mon[22428]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:10.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:09 smithi077 ceph-mon[22428]: from='client.25090 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:43:10.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:09 smithi170 ceph-mon[23272]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:10.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:09 smithi170 ceph-mon[23272]: from='client.25090 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:43:12.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:11 smithi077 ceph-mon[27166]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:43:12.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:11 smithi077 ceph-mon[22428]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:43:12.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:11 smithi170 ceph-mon[23272]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:43:14.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:13 smithi077 ceph-mon[22428]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:14.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:13 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:43:14.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:13 smithi077 ceph-mon[27166]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:14.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:13 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:43:14.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:13 smithi170 ceph-mon[23272]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:14.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:13 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:43:16.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:15 smithi077 ceph-mon[22428]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:43:16.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:15 smithi077 ceph-mon[27166]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:43:16.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:15 smithi170 ceph-mon[23272]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:43:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:17 smithi077 ceph-mon[22428]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:18.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:17 smithi077 ceph-mon[27166]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:18.317 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:17 smithi170 ceph-mon[23272]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:19.046 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:43:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:43:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:43:19.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:19 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:43:19] "GET /metrics HTTP/1.1" 200 37897 "" "Prometheus/2.33.4" 2024-05-06T13:43:20.091 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:19 smithi170 ceph-mon[23272]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:20.091 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:19 smithi170 ceph-mon[23272]: from='client.25090 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:43:20.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:19 smithi077 ceph-mon[22428]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:20.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:19 smithi077 ceph-mon[22428]: from='client.25090 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:43:20.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:19 smithi077 ceph-mon[27166]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:20.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:19 smithi077 ceph-mon[27166]: from='client.25090 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:43:22.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:21 smithi077 ceph-mon[22428]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:43:22.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:21 smithi077 ceph-mon[27166]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:43:22.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:21 smithi170 ceph-mon[23272]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:43:23.129 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:22 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:43:23.130 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:22 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:43:23.130 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:22 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:43:23.130 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:22 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:43:23.130 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:22 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:43:23.130 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:22 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:43:23.130 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:22 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:43:23.130 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:22 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:43:23.131 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:22 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:43:23.131 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:22 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:43:23.131 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:22 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:43:23.131 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:22 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:43:23.131 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:22 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:43:23.131 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:22 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:43:23.131 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:22 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:43:23.131 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:22 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:43:23.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:22 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:43:23.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:22 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:43:23.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:22 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:43:23.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:22 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:43:23.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:22 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:43:23.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:22 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:43:23.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:22 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:43:23.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:22 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:43:24.143 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:23 smithi077 ceph-mon[22428]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:24.143 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:23 smithi077 ceph-mon[22428]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:43:24.143 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:23 smithi077 ceph-mon[22428]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:43:24.143 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:23 smithi077 ceph-mon[27166]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:24.143 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:23 smithi077 ceph-mon[27166]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:43:24.143 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:23 smithi077 ceph-mon[27166]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:43:24.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:23 smithi170 ceph-mon[23272]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:24.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:23 smithi170 ceph-mon[23272]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:43:24.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:23 smithi170 ceph-mon[23272]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:43:26.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:25 smithi170 ceph-mon[23272]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-06T13:43:26.076 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:43:25.680+0000 7f893079f700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:43:26.076 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Creating ceph-iscsi config... 2024-05-06T13:43:26.077 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:43:26.077 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:43:26.077 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:43:26.077 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:43:26.077 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:43:26.077 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Traceback (most recent call last): 2024-05-06T13:43:26.077 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:43:26.077 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return _run_code(code, main_globals, None, 2024-05-06T13:43:26.077 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:43:26.077 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: exec(code, run_globals) 2024-05-06T13:43:26.077 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:43:26.077 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:43:26.078 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:43:26.078 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:43:26.078 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:43:26.078 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:43:26.078 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:43:26.078 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:43:26.079 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:43:26.079 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Traceback (most recent call last): 2024-05-06T13:43:26.079 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:43:26.079 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:43:26.079 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:43:26.079 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:43:26.079 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:43:26.079 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:43:26.079 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:43:26.079 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return future.result(timeout) 2024-05-06T13:43:26.079 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:43:26.079 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return self.__get_result() 2024-05-06T13:43:26.080 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:43:26.080 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: raise self._exception 2024-05-06T13:43:26.080 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:43:26.080 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: use_current_daemon_image=reconfig, 2024-05-06T13:43:26.080 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:43:26.080 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:43:26.080 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:43:26.080 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Creating ceph-iscsi config... 2024-05-06T13:43:26.080 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:43:26.080 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:43:26.080 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:43:26.081 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:43:26.081 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:43:26.081 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Traceback (most recent call last): 2024-05-06T13:43:26.081 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:43:26.081 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return _run_code(code, main_globals, None, 2024-05-06T13:43:26.081 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:43:26.081 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: exec(code, run_globals) 2024-05-06T13:43:26.081 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:43:26.081 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:43:26.081 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:43:26.081 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:43:26.081 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:43:26.081 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:43:26.081 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:43:26.082 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:43:26.082 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:43:26.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:25 smithi077 ceph-mon[22428]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-06T13:43:26.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:25 smithi077 ceph-mon[27166]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-06T13:43:27.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:26 smithi077 ceph-mon[22428]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:43:27.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:43:27.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:43:27.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:43:27.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:43:27.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:43:27.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:43:27.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:43:27.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:43:27.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:43:27.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:43:27.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:43:27.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:43:27.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:43:27.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:43:27.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:43:27.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:43:27.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:43:27.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:43:27.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:43:27.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:43:27.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:43:27.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:43:27.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:43:27.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:43:27.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:43:27.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:43:27.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:43:27.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:43:27.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: return future.result(timeout) 2024-05-06T13:43:27.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:43:27.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: return self.__get_result() 2024-05-06T13:43:27.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:43:27.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: raise self._exception 2024-05-06T13:43:27.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:43:27.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: use_current_daemon_image=reconfig, 2024-05-06T13:43:27.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:43:27.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:43:27.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:43:27.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:43:27.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:43:27.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:43:27.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:43:27.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:43:27.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:43:27.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:43:27.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:43:27.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:43:27.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:43:27.057 INFO:journalctl@ceph.mon.a.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:43:27.057 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:43:27.057 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:43:27.057 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:43:27.057 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:43:27.057 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:43:27.057 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:43:27.057 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:43:27.057 INFO:journalctl@ceph.mon.a.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:43:27.057 INFO:journalctl@ceph.mon.a.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:43:27.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:26 smithi077 ceph-mon[27166]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:43:27.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:43:27.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:43:27.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:43:27.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:43:27.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:43:27.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:43:27.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:43:27.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:43:27.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:43:27.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:43:27.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:43:27.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:43:27.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:43:27.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:43:27.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:43:27.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:43:27.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:43:27.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:43:27.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:43:27.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:43:27.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:43:27.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:43:27.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:43:27.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:43:27.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:43:27.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:43:27.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:43:27.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:43:27.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: return future.result(timeout) 2024-05-06T13:43:27.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:43:27.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: return self.__get_result() 2024-05-06T13:43:27.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:43:27.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: raise self._exception 2024-05-06T13:43:27.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:43:27.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: use_current_daemon_image=reconfig, 2024-05-06T13:43:27.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:43:27.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:43:27.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:43:27.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:43:27.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:43:27.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:43:27.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:43:27.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:43:27.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:43:27.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:43:27.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:43:27.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:43:27.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:43:27.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:43:27.062 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:43:27.062 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:43:27.062 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:43:27.062 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:43:27.062 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:43:27.062 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:43:27.062 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:43:27.062 INFO:journalctl@ceph.mon.c.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:43:27.062 INFO:journalctl@ceph.mon.c.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:43:27.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:26 smithi170 ceph-mon[23272]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:43:27.326 INFO:journalctl@ceph.mon.b.smithi170.stdout: Creating ceph-iscsi config... 2024-05-06T13:43:27.326 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:43:27.327 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:43:27.327 INFO:journalctl@ceph.mon.b.smithi170.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:43:27.327 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:43:27.327 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:43:27.327 INFO:journalctl@ceph.mon.b.smithi170.stdout: Traceback (most recent call last): 2024-05-06T13:43:27.327 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:43:27.327 INFO:journalctl@ceph.mon.b.smithi170.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:43:27.327 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:43:27.327 INFO:journalctl@ceph.mon.b.smithi170.stdout: exec(code, run_globals) 2024-05-06T13:43:27.327 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:43:27.327 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:43:27.327 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:43:27.327 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:43:27.327 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:43:27.328 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:43:27.328 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:43:27.328 INFO:journalctl@ceph.mon.b.smithi170.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:43:27.328 INFO:journalctl@ceph.mon.b.smithi170.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:43:27.328 INFO:journalctl@ceph.mon.b.smithi170.stdout: Traceback (most recent call last): 2024-05-06T13:43:27.328 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:43:27.328 INFO:journalctl@ceph.mon.b.smithi170.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:43:27.328 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:43:27.328 INFO:journalctl@ceph.mon.b.smithi170.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:43:27.328 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:43:27.328 INFO:journalctl@ceph.mon.b.smithi170.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:43:27.328 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:43:27.328 INFO:journalctl@ceph.mon.b.smithi170.stdout: return future.result(timeout) 2024-05-06T13:43:27.328 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:43:27.328 INFO:journalctl@ceph.mon.b.smithi170.stdout: return self.__get_result() 2024-05-06T13:43:27.329 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:43:27.329 INFO:journalctl@ceph.mon.b.smithi170.stdout: raise self._exception 2024-05-06T13:43:27.329 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:43:27.329 INFO:journalctl@ceph.mon.b.smithi170.stdout: use_current_daemon_image=reconfig, 2024-05-06T13:43:27.329 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:43:27.329 INFO:journalctl@ceph.mon.b.smithi170.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:43:27.329 INFO:journalctl@ceph.mon.b.smithi170.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:43:27.329 INFO:journalctl@ceph.mon.b.smithi170.stdout: Creating ceph-iscsi config... 2024-05-06T13:43:27.329 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:43:27.329 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:43:27.329 INFO:journalctl@ceph.mon.b.smithi170.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:43:27.329 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:43:27.329 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:43:27.329 INFO:journalctl@ceph.mon.b.smithi170.stdout: Traceback (most recent call last): 2024-05-06T13:43:27.329 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:43:27.330 INFO:journalctl@ceph.mon.b.smithi170.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:43:27.330 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:43:27.330 INFO:journalctl@ceph.mon.b.smithi170.stdout: exec(code, run_globals) 2024-05-06T13:43:27.330 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:43:27.330 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:43:27.330 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:43:27.330 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:43:27.330 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:43:27.330 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:43:27.330 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:43:27.330 INFO:journalctl@ceph.mon.b.smithi170.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:43:27.330 INFO:journalctl@ceph.mon.b.smithi170.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:43:28.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:27 smithi077 ceph-mon[22428]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-05-06T13:43:28.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:27 smithi077 ceph-mon[27166]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-05-06T13:43:28.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:27 smithi170 ceph-mon[23272]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-05-06T13:43:29.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:28 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:43:29.076 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:43:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:43:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:43:29.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:28 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:43:29.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:28 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:43:29.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:43:29] "GET /metrics HTTP/1.1" 200 37897 "" "Prometheus/2.33.4" 2024-05-06T13:43:30.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:29 smithi077 ceph-mon[22428]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-05-06T13:43:30.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:29 smithi077 ceph-mon[27166]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-05-06T13:43:30.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:29 smithi170 ceph-mon[23272]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-05-06T13:43:31.520 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-05-06T13:43:32.086 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:31 smithi077 ceph-mon[22428]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-05-06T13:43:32.087 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:31 smithi077 ceph-mon[27166]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-05-06T13:43:32.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:31 smithi170 ceph-mon[23272]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-05-06T13:43:33.935 INFO:teuthology.orchestra.run.smithi077.stdout:Scheduled to redeploy mgr.y on host 'smithi077' 2024-05-06T13:43:34.273 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:33 smithi077 ceph-mon[22428]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-06T13:43:34.273 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:33 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2634746997' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-06T13:43:34.273 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:33 smithi077 ceph-mon[27166]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-06T13:43:34.273 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:33 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2634746997' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-06T13:43:34.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:33 smithi170 ceph-mon[23272]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-06T13:43:34.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:33 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2634746997' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-06T13:43:34.580 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch ps --refresh' 2024-05-06T13:43:35.194 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:34 smithi077 ceph-mon[22428]: from='client.15207 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:43:35.194 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:34 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:43:35.194 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:34 smithi077 ceph-mon[22428]: Schedule redeploy daemon mgr.y 2024-05-06T13:43:35.195 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:34 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:43:35.195 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:34 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:43:35.195 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:34 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:43:35.195 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:34 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:43:35.195 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:34 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:43:35.195 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:34 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T13:43:35.195 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:34 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T13:43:35.195 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:34 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T13:43:35.195 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:34 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:43:35.195 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:34 smithi077 ceph-mon[27166]: from='client.15207 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:43:35.195 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:34 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:43:35.195 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:34 smithi077 ceph-mon[27166]: Schedule redeploy daemon mgr.y 2024-05-06T13:43:35.195 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:34 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:43:35.196 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:34 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:43:35.196 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:34 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:43:35.196 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:34 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:43:35.196 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:34 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:43:35.196 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:34 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T13:43:35.196 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:34 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T13:43:35.196 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:34 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T13:43:35.196 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:34 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:43:35.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:34 smithi170 ceph-mon[23272]: from='client.15207 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:43:35.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:34 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:43:35.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:34 smithi170 ceph-mon[23272]: Schedule redeploy daemon mgr.y 2024-05-06T13:43:35.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:34 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:43:35.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:34 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:43:35.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:34 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:43:35.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:34 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:43:35.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:34 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:43:35.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:34 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T13:43:35.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:34 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T13:43:35.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:34 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T13:43:35.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:34 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:43:36.254 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:35 smithi077 ceph-mon[22428]: Deploying daemon mgr.y on smithi077 2024-05-06T13:43:36.254 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:35 smithi077 ceph-mon[22428]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-06T13:43:36.254 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:35 smithi077 ceph-mon[27166]: Deploying daemon mgr.y on smithi077 2024-05-06T13:43:36.254 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:35 smithi077 ceph-mon[27166]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-06T13:43:36.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:35 smithi170 ceph-mon[23272]: Deploying daemon mgr.y on smithi077 2024-05-06T13:43:36.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:35 smithi170 ceph-mon[23272]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-06T13:43:36.769 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T13:43:36.769 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (2m) 83s ago 10m 17.5M - 0.23.0 ba2b418f427c 0f855548777f 2024-05-06T13:43:36.769 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (2m) 14s ago 10m 47.4M - 8.3.5 dad864ee21e9 fdcecac79925 2024-05-06T13:43:36.769 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (2m) 83s ago 9m 46.1M - 3.5 e1d6a67b021e 49f2e7c1d2f1 2024-05-06T13:43:36.769 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283 running (6m) 14s ago 15m 511M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 63ef6825ff68 2024-05-06T13:43:36.769 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:9283 running (16m) 83s ago 16m 413M - 17.2.0 e1d6a67b021e 79a55d163d05 2024-05-06T13:43:36.769 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (16m) 83s ago 16m 54.5M 2048M 17.2.0 e1d6a67b021e ac46f91694f8 2024-05-06T13:43:36.769 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (15m) 14s ago 15m 45.4M 2048M 17.2.0 e1d6a67b021e 24be6fce9d28 2024-05-06T13:43:36.770 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (15m) 83s ago 15m 43.0M 2048M 17.2.0 e1d6a67b021e 83af336ef762 2024-05-06T13:43:36.770 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (2m) 83s ago 11m 17.7M - 1.3.1 1dbe0e931976 54aee5b22b1c 2024-05-06T13:43:36.770 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (2m) 14s ago 11m 17.2M - 1.3.1 1dbe0e931976 ca5df43a7469 2024-05-06T13:43:36.770 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (14m) 83s ago 14m 51.4M 2992M 17.2.0 e1d6a67b021e 2ccee2bb6e96 2024-05-06T13:43:36.770 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (14m) 83s ago 14m 55.2M 2992M 17.2.0 e1d6a67b021e 1d9e6609b585 2024-05-06T13:43:36.770 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (14m) 83s ago 14m 49.8M 2992M 17.2.0 e1d6a67b021e 37a61d295111 2024-05-06T13:43:36.770 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (13m) 83s ago 13m 50.5M 2992M 17.2.0 e1d6a67b021e 8feb7c077aaa 2024-05-06T13:43:36.770 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (13m) 14s ago 13m 53.6M 3248M 17.2.0 e1d6a67b021e f865acc44a06 2024-05-06T13:43:36.770 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (13m) 14s ago 13m 50.1M 3248M 17.2.0 e1d6a67b021e a55bd4122f6f 2024-05-06T13:43:36.770 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (12m) 14s ago 12m 48.6M 3248M 17.2.0 e1d6a67b021e 362938a0b357 2024-05-06T13:43:36.770 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (12m) 14s ago 12m 50.8M 3248M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T13:43:36.770 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (2m) 14s ago 11m 64.3M - 2.33.4 514e6a882f6e c45c7b7e4059 2024-05-06T13:43:36.770 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (10m) 83s ago 10m 73.4M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T13:43:36.771 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (9m) 14s ago 9m 72.3M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T13:43:37.561 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'sleep 180' 2024-05-06T13:43:38.181 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:37 smithi077 ceph-mon[22428]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:43:38.181 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:37 smithi077 ceph-mon[22428]: from='client.15213 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:43:38.181 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:37 smithi077 ceph-mon[27166]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:43:38.181 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:37 smithi077 ceph-mon[27166]: from='client.15213 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:43:38.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:37 smithi170 ceph-mon[23272]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:43:38.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:37 smithi170 ceph-mon[23272]: from='client.15213 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:43:39.076 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:43:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:43:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:43:39.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:39 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:43:39] "GET /metrics HTTP/1.1" 200 37897 "" "Prometheus/2.33.4" 2024-05-06T13:43:40.258 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:39 smithi077 ceph-mon[22428]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:43:40.258 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:39 smithi077 ceph-mon[27166]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:43:40.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:39 smithi170 ceph-mon[23272]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:43:41.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:40 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2158699080' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:43:41.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:40 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2158699080' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:43:41.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:40 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2158699080' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:43:42.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:41 smithi077 ceph-mon[22428]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-05-06T13:43:42.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:41 smithi077 ceph-mon[27166]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-05-06T13:43:42.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:41 smithi170 ceph-mon[23272]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-05-06T13:43:44.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:43 smithi077 ceph-mon[22428]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-05-06T13:43:44.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:43 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:43:44.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:43 smithi077 ceph-mon[27166]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-05-06T13:43:44.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:43 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:43:44.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:43 smithi170 ceph-mon[23272]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-05-06T13:43:44.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:43 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:43:46.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:45 smithi077 ceph-mon[22428]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-05-06T13:43:46.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:45 smithi077 ceph-mon[27166]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-05-06T13:43:46.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:45 smithi170 ceph-mon[23272]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-05-06T13:43:48.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:48 smithi077 ceph-mon[22428]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-05-06T13:43:48.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:48 smithi077 ceph-mon[27166]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-05-06T13:43:48.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:48 smithi170 ceph-mon[23272]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-05-06T13:43:49.076 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:43:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:43:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:43:49.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:49 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:43:49] "GET /metrics HTTP/1.1" 200 37904 "" "Prometheus/2.33.4" 2024-05-06T13:43:50.280 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:50 smithi077 ceph-mon[22428]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-05-06T13:43:50.280 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:50 smithi077 ceph-mon[27166]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-05-06T13:43:50.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:50 smithi170 ceph-mon[23272]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-05-06T13:43:52.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:52 smithi077 ceph-mon[22428]: from='client.25135 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:43:52.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:52 smithi077 ceph-mon[22428]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 120 op/s 2024-05-06T13:43:52.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:52 smithi077 ceph-mon[27166]: from='client.25135 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:43:52.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:52 smithi077 ceph-mon[27166]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 120 op/s 2024-05-06T13:43:52.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:52 smithi170 ceph-mon[23272]: from='client.25135 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:43:52.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:52 smithi170 ceph-mon[23272]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 120 op/s 2024-05-06T13:43:54.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:54 smithi077 ceph-mon[27166]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:54.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:54 smithi077 ceph-mon[22428]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:54.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:54 smithi170 ceph-mon[23272]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:56.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:56 smithi077 ceph-mon[22428]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:43:56.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:56 smithi077 ceph-mon[27166]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:43:56.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:56 smithi170 ceph-mon[23272]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:43:58.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:58 smithi077 ceph-mon[22428]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:58.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:58 smithi077 ceph-mon[27166]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:58.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:58 smithi170 ceph-mon[23272]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:43:59.018 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:43:58 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:43:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:43:59.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:43:59 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:43:59.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:43:59 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:43:59.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:43:59 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:43:59.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:43:59 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:43:59] "GET /metrics HTTP/1.1" 200 37904 "" "Prometheus/2.33.4" 2024-05-06T13:44:00.291 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:00 smithi077 ceph-mon[22428]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:00.291 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:00 smithi077 ceph-mon[27166]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:00.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:00 smithi170 ceph-mon[23272]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:02.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:02 smithi077 ceph-mon[22428]: from='client.25135 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:44:02.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:02 smithi077 ceph-mon[22428]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:44:02.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:02 smithi077 ceph-mon[27166]: from='client.25135 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:44:02.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:02 smithi077 ceph-mon[27166]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:44:02.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:02 smithi170 ceph-mon[23272]: from='client.25135 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:44:02.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:02 smithi170 ceph-mon[23272]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:44:04.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:04 smithi077 ceph-mon[22428]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:04.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:04 smithi077 ceph-mon[27166]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:04.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:04 smithi170 ceph-mon[23272]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:06.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:06 smithi077 ceph-mon[22428]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:44:06.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:06 smithi077 ceph-mon[27166]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:44:06.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:06 smithi170 ceph-mon[23272]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:44:08.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:08 smithi077 ceph-mon[22428]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:08.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:08 smithi077 ceph-mon[27166]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:08.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:08 smithi170 ceph-mon[23272]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:09.076 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:44:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:44:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:44:09.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:44:09] "GET /metrics HTTP/1.1" 200 37904 "" "Prometheus/2.33.4" 2024-05-06T13:44:10.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:10 smithi077 ceph-mon[22428]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:10.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:10 smithi077 ceph-mon[27166]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:10.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:10 smithi170 ceph-mon[23272]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:11.053 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:10 smithi077 systemd[1]: Stopping Ceph mgr.y for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:44:11.396 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:11 smithi077 podman[60613]: 2024-05-06 13:44:11.069735878 +0000 UTC m=+0.133502440 container died 79a55d163d052c84d979db49ef7f91554cf70486bb677633de3378378d731f8d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, vcs-type=git, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, release=754, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., ceph=True, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, architecture=x86_64, distribution-scope=public, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-05-06T13:44:11.798 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:11 smithi077 podman[60613]: 2024-05-06 13:44:11.397038432 +0000 UTC m=+0.460805001 container cleanup 79a55d163d052c84d979db49ef7f91554cf70486bb677633de3378378d731f8d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, release=754, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, RELEASE=HEAD, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, vcs-type=git, build-date=2022-05-03T08:36:31.336870, name=centos-stream, version=8, distribution-scope=public, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-05-06T13:44:11.799 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:11 smithi077 bash[60613]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y 2024-05-06T13:44:11.799 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:11 smithi077 podman[60627]: 2024-05-06 13:44:11.581898165 +0000 UTC m=+0.516978122 container remove 79a55d163d052c84d979db49ef7f91554cf70486bb677633de3378378d731f8d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y, build-date=2022-05-03T08:36:31.336870, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., name=centos-stream, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, io.buildah.version=1.19.8, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-05-06T13:44:11.799 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:11 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-05-06T13:44:12.240 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:12 smithi077 ceph-mon[22428]: from='client.25135 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:44:12.240 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:12 smithi077 ceph-mon[22428]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:44:12.241 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:12 smithi077 ceph-mon[27166]: from='client.25135 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:44:12.241 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:12 smithi077 ceph-mon[27166]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:44:12.241 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:11 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-05-06T13:44:12.241 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:11 smithi077 systemd[1]: Stopped Ceph mgr.y for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:44:12.241 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:11 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mgr.y.service: Consumed 59.710s CPU time. 2024-05-06T13:44:12.241 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:12 smithi077 systemd[1]: Starting Ceph mgr.y for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:44:12.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:12 smithi170 ceph-mon[23272]: from='client.25135 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:44:12.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:12 smithi170 ceph-mon[23272]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:44:12.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:12 smithi077 podman[60762]: 2024-05-06 13:44:12.143668453 +0000 UTC m=+0.017832947 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:44:12.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:12 smithi077 podman[60762]: 2024-05-06 13:44:12.306698697 +0000 UTC m=+0.180863190 container create 129a14f19c525bdbf648f43c0353e1dd88fe1c91e2d821bf08c1e2b5ea186f7b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-05-06T13:44:12.929 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:12 smithi077 podman[60762]: 2024-05-06 13:44:12.574652418 +0000 UTC m=+0.448816913 container init 129a14f19c525bdbf648f43c0353e1dd88fe1c91e2d821bf08c1e2b5ea186f7b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-70ce953, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-05-06T13:44:12.929 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:12 smithi077 podman[60762]: 2024-05-06 13:44:12.577423233 +0000 UTC m=+0.451587731 container start 129a14f19c525bdbf648f43c0353e1dd88fe1c91e2d821bf08c1e2b5ea186f7b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-05-06T13:44:12.929 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:12 smithi077 ceph-mgr[60780]: -- 172.21.15.77:0/3500398969 <== mon.1 v2:172.21.15.77:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cb07ff85a0 con 0x55cb07fcc800 2024-05-06T13:44:12.929 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:12 smithi077 bash[60762]: 129a14f19c525bdbf648f43c0353e1dd88fe1c91e2d821bf08c1e2b5ea186f7b 2024-05-06T13:44:12.929 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:12 smithi077 systemd[1]: Started Ceph mgr.y for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:44:12.929 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:44:12.729+0000 7f5daeb65200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T13:44:12.929 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:44:12.927+0000 7f5daeb65200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:44:13.302 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:44:13.036+0000 7f5daeb65200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T13:44:13.665 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:44:13.374+0000 7f5daeb65200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T13:44:13.665 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:44:13.462+0000 7f5daeb65200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T13:44:14.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:13 smithi077 ceph-mon[22428]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:14.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:13 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:14.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:13 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:14.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:13 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:14.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:13 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:14.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:13 smithi077 ceph-mon[22428]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:44:14.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:13 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:44:14.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:13 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:44:14.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:13 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:44:14.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:13 smithi077 ceph-mon[22428]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:44:14.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:13 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:44:14.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:13 smithi077 ceph-mon[27166]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:14.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:13 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:14.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:13 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:14.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:13 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:14.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:13 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:14.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:13 smithi077 ceph-mon[27166]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:44:14.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:13 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:44:14.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:13 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:44:14.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:13 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:44:14.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:13 smithi077 ceph-mon[27166]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:44:14.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:13 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:44:14.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:13 smithi170 ceph-mon[23272]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:14.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:13 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:14.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:13 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:14.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:13 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:14.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:13 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:14.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:13 smithi170 ceph-mon[23272]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:44:14.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:13 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:44:14.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:13 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:44:14.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:13 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:44:14.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:13 smithi170 ceph-mon[23272]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:44:14.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:13 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:44:15.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:14 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:44:14.717+0000 7f5daeb65200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T13:44:15.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:14 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:44:14.892+0000 7f5daeb65200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T13:44:15.359 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:15 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:44:15.067+0000 7f5daeb65200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T13:44:15.667 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:15 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:44:15.358+0000 7f5daeb65200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T13:44:15.668 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:15 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:44:15.455+0000 7f5daeb65200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T13:44:16.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:15 smithi077 ceph-mon[22428]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:44:16.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:15 smithi077 ceph-mon[27166]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:44:16.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:15 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:44:15.667+0000 7f5daeb65200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T13:44:16.053 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:15 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:44:15.864+0000 7f5daeb65200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:44:16.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:15 smithi170 ceph-mon[23272]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:44:16.802 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:16 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:44:16.477+0000 7f5daeb65200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T13:44:17.241 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:16 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:44:16.932+0000 7f5daeb65200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T13:44:17.241 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:17 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:44:17.033+0000 7f5daeb65200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T13:44:17.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:17 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:44:17.240+0000 7f5daeb65200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-06T13:44:18.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:17 smithi077 ceph-mon[22428]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:18.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:17 smithi077 ceph-mon[27166]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:18.053 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:17 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:44:17.784+0000 7f5daeb65200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T13:44:18.053 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:17 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:44:17.882+0000 7f5daeb65200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T13:44:18.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:17 smithi170 ceph-mon[23272]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:18.322 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:18 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:44:18.142+0000 7f5daeb65200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T13:44:18.323 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:18 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:44:18.281+0000 7f5daeb65200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T13:44:18.613 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:18 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:44:18.382+0000 7f5daeb65200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T13:44:18.613 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:18 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:44:18.507+0000 7f5daeb65200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T13:44:18.613 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:18 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:44:18.611+0000 7f5daeb65200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T13:44:18.826 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:44:18.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:44:18.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:44:18.428+0000 7f893079f700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:44:18.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Creating ceph-iscsi config... 2024-05-06T13:44:18.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:44:18.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:44:18.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:44:18.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:44:18.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:44:18.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Traceback (most recent call last): 2024-05-06T13:44:18.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:44:18.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return _run_code(code, main_globals, None, 2024-05-06T13:44:18.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:44:18.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: exec(code, run_globals) 2024-05-06T13:44:18.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:44:18.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:44:18.828 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:44:18.828 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:44:18.828 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:44:18.828 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:44:18.828 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:44:18.828 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:44:18.828 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:44:18.828 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Traceback (most recent call last): 2024-05-06T13:44:18.828 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:44:18.828 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:44:18.828 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:44:18.828 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:44:18.828 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:44:18.828 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:44:18.828 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:44:18.828 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return future.result(timeout) 2024-05-06T13:44:18.829 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:44:18.829 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return self.__get_result() 2024-05-06T13:44:18.829 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:44:18.829 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: raise self._exception 2024-05-06T13:44:18.829 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:44:18.829 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: use_current_daemon_image=reconfig, 2024-05-06T13:44:18.829 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:44:18.829 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:44:18.829 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:44:18.829 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Creating ceph-iscsi config... 2024-05-06T13:44:18.829 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:44:18.829 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:44:18.829 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:44:18.829 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:44:18.830 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:44:18.830 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Traceback (most recent call last): 2024-05-06T13:44:18.830 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:44:18.830 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return _run_code(code, main_globals, None, 2024-05-06T13:44:18.830 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:44:18.830 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: exec(code, run_globals) 2024-05-06T13:44:18.830 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:44:18.830 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:44:18.830 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:44:18.830 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:44:18.831 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:44:18.831 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:44:18.831 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:44:18.831 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:44:18.831 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:44:19.478 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:19 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:44:19] "GET /metrics HTTP/1.1" 200 37900 "" "Prometheus/2.33.4" 2024-05-06T13:44:19.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:19 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:44:19.491+0000 7f5daeb65200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T13:44:19.765 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:19 smithi170 ceph-mon[23272]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:19.765 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:19 smithi170 ceph-mon[23272]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:44:19.765 INFO:journalctl@ceph.mon.b.smithi170.stdout: Creating ceph-iscsi config... 2024-05-06T13:44:19.765 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:44:19.765 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:44:19.765 INFO:journalctl@ceph.mon.b.smithi170.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:44:19.765 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:44:19.766 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:44:19.766 INFO:journalctl@ceph.mon.b.smithi170.stdout: Traceback (most recent call last): 2024-05-06T13:44:19.766 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:44:19.766 INFO:journalctl@ceph.mon.b.smithi170.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:44:19.766 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:44:19.766 INFO:journalctl@ceph.mon.b.smithi170.stdout: exec(code, run_globals) 2024-05-06T13:44:19.766 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:44:19.766 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:44:19.766 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:44:19.766 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:44:19.766 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:44:19.766 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:44:19.766 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:44:19.766 INFO:journalctl@ceph.mon.b.smithi170.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:44:19.767 INFO:journalctl@ceph.mon.b.smithi170.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:44:19.767 INFO:journalctl@ceph.mon.b.smithi170.stdout: Traceback (most recent call last): 2024-05-06T13:44:19.767 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:44:19.767 INFO:journalctl@ceph.mon.b.smithi170.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:44:19.767 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:44:19.767 INFO:journalctl@ceph.mon.b.smithi170.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:44:19.767 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:44:19.767 INFO:journalctl@ceph.mon.b.smithi170.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:44:19.767 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:44:19.767 INFO:journalctl@ceph.mon.b.smithi170.stdout: return future.result(timeout) 2024-05-06T13:44:19.767 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:44:19.767 INFO:journalctl@ceph.mon.b.smithi170.stdout: return self.__get_result() 2024-05-06T13:44:19.767 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:44:19.767 INFO:journalctl@ceph.mon.b.smithi170.stdout: raise self._exception 2024-05-06T13:44:19.767 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:44:19.767 INFO:journalctl@ceph.mon.b.smithi170.stdout: use_current_daemon_image=reconfig, 2024-05-06T13:44:19.768 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:44:19.768 INFO:journalctl@ceph.mon.b.smithi170.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:44:19.768 INFO:journalctl@ceph.mon.b.smithi170.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:44:19.768 INFO:journalctl@ceph.mon.b.smithi170.stdout: Creating ceph-iscsi config... 2024-05-06T13:44:19.768 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:44:19.768 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:44:19.768 INFO:journalctl@ceph.mon.b.smithi170.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:44:19.768 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:44:19.768 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:44:19.768 INFO:journalctl@ceph.mon.b.smithi170.stdout: Traceback (most recent call last): 2024-05-06T13:44:19.768 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:44:19.768 INFO:journalctl@ceph.mon.b.smithi170.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:44:19.768 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:44:19.768 INFO:journalctl@ceph.mon.b.smithi170.stdout: exec(code, run_globals) 2024-05-06T13:44:19.768 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:44:19.769 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:44:19.769 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:44:19.769 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:44:19.769 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:44:19.769 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:44:19.769 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:44:19.769 INFO:journalctl@ceph.mon.b.smithi170.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:44:19.769 INFO:journalctl@ceph.mon.b.smithi170.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:44:19.769 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:19 smithi170 ceph-mon[23272]: Standby manager daemon y restarted 2024-05-06T13:44:19.769 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:19 smithi170 ceph-mon[23272]: Standby manager daemon y started 2024-05-06T13:44:19.769 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:19 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-06T13:44:19.769 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:19 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:44:19.769 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:19 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-06T13:44:19.769 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:19 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:44:20.021 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:19 smithi077 ceph-mon[22428]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:20.021 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:19 smithi077 ceph-mon[22428]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:44:20.021 INFO:journalctl@ceph.mon.a.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:44:20.021 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:44:20.021 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:44:20.021 INFO:journalctl@ceph.mon.a.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:44:20.021 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:44:20.021 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:44:20.021 INFO:journalctl@ceph.mon.a.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:44:20.021 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:44:20.021 INFO:journalctl@ceph.mon.a.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:44:20.021 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:44:20.022 INFO:journalctl@ceph.mon.a.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:44:20.022 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:44:20.022 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:44:20.022 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:44:20.022 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:44:20.022 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:44:20.022 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:44:20.022 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:44:20.022 INFO:journalctl@ceph.mon.a.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:44:20.022 INFO:journalctl@ceph.mon.a.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:44:20.022 INFO:journalctl@ceph.mon.a.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:44:20.022 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:44:20.022 INFO:journalctl@ceph.mon.a.smithi077.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:44:20.022 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:44:20.023 INFO:journalctl@ceph.mon.a.smithi077.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:44:20.023 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:44:20.023 INFO:journalctl@ceph.mon.a.smithi077.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:44:20.023 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:44:20.023 INFO:journalctl@ceph.mon.a.smithi077.stdout: return future.result(timeout) 2024-05-06T13:44:20.023 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:44:20.023 INFO:journalctl@ceph.mon.a.smithi077.stdout: return self.__get_result() 2024-05-06T13:44:20.023 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:44:20.023 INFO:journalctl@ceph.mon.a.smithi077.stdout: raise self._exception 2024-05-06T13:44:20.023 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:44:20.023 INFO:journalctl@ceph.mon.a.smithi077.stdout: use_current_daemon_image=reconfig, 2024-05-06T13:44:20.023 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:44:20.023 INFO:journalctl@ceph.mon.a.smithi077.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:44:20.023 INFO:journalctl@ceph.mon.a.smithi077.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:44:20.023 INFO:journalctl@ceph.mon.a.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:44:20.024 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:44:20.024 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:44:20.024 INFO:journalctl@ceph.mon.a.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:44:20.024 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:44:20.024 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:44:20.024 INFO:journalctl@ceph.mon.a.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:44:20.024 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:44:20.024 INFO:journalctl@ceph.mon.a.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:44:20.024 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:44:20.024 INFO:journalctl@ceph.mon.a.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:44:20.024 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:44:20.024 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:44:20.025 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:44:20.025 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:44:20.025 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:19 smithi077 ceph-mon[27166]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:20.025 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:19 smithi077 ceph-mon[27166]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:44:20.025 INFO:journalctl@ceph.mon.c.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:44:20.025 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:44:20.025 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:44:20.025 INFO:journalctl@ceph.mon.c.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:44:20.025 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:44:20.025 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:44:20.026 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:44:20.026 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:44:20.026 INFO:journalctl@ceph.mon.c.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:44:20.026 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:44:20.026 INFO:journalctl@ceph.mon.c.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:44:20.026 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:44:20.026 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:44:20.026 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:44:20.026 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:44:20.026 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:44:20.026 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:44:20.026 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:44:20.026 INFO:journalctl@ceph.mon.c.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:44:20.026 INFO:journalctl@ceph.mon.c.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:44:20.027 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:44:20.027 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:44:20.027 INFO:journalctl@ceph.mon.c.smithi077.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:44:20.027 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:44:20.027 INFO:journalctl@ceph.mon.c.smithi077.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:44:20.027 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:44:20.027 INFO:journalctl@ceph.mon.c.smithi077.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:44:20.027 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:44:20.027 INFO:journalctl@ceph.mon.c.smithi077.stdout: return future.result(timeout) 2024-05-06T13:44:20.027 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:44:20.027 INFO:journalctl@ceph.mon.c.smithi077.stdout: return self.__get_result() 2024-05-06T13:44:20.027 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:44:20.027 INFO:journalctl@ceph.mon.c.smithi077.stdout: raise self._exception 2024-05-06T13:44:20.027 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:44:20.027 INFO:journalctl@ceph.mon.c.smithi077.stdout: use_current_daemon_image=reconfig, 2024-05-06T13:44:20.028 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:44:20.028 INFO:journalctl@ceph.mon.c.smithi077.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:44:20.028 INFO:journalctl@ceph.mon.c.smithi077.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:44:20.028 INFO:journalctl@ceph.mon.c.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:44:20.028 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:44:20.028 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:44:20.028 INFO:journalctl@ceph.mon.c.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:44:20.028 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:44:20.028 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:44:20.028 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:44:20.028 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:44:20.028 INFO:journalctl@ceph.mon.c.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:44:20.028 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:44:20.028 INFO:journalctl@ceph.mon.c.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:44:20.028 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:44:20.029 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:44:20.029 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:44:20.029 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:44:20.029 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:44:20.029 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:44:20.029 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:44:20.029 INFO:journalctl@ceph.mon.a.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:44:20.029 INFO:journalctl@ceph.mon.a.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:44:20.029 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:19 smithi077 ceph-mon[22428]: Standby manager daemon y restarted 2024-05-06T13:44:20.029 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:19 smithi077 ceph-mon[22428]: Standby manager daemon y started 2024-05-06T13:44:20.029 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:19 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-06T13:44:20.029 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:19 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:44:20.030 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:19 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-06T13:44:20.030 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:19 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:44:20.030 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:19 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:44:19.715+0000 7f5daeb65200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T13:44:20.030 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:19 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:44:19] ENGINE Bus STARTING 2024-05-06T13:44:20.030 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:19 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: CherryPy Checker: 2024-05-06T13:44:20.030 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:19 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: The Application mounted at '' has an empty config. 2024-05-06T13:44:20.030 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:19 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:44:20.030 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:19 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:44:19] ENGINE Serving on http://:::9283 2024-05-06T13:44:20.030 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:44:19 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:44:19] ENGINE Bus STARTED 2024-05-06T13:44:20.031 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:44:20.031 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:44:20.031 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:44:20.031 INFO:journalctl@ceph.mon.c.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:44:20.031 INFO:journalctl@ceph.mon.c.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:44:20.031 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:19 smithi077 ceph-mon[27166]: Standby manager daemon y restarted 2024-05-06T13:44:20.031 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:19 smithi077 ceph-mon[27166]: Standby manager daemon y started 2024-05-06T13:44:20.031 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:19 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-06T13:44:20.031 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:19 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:44:20.031 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:19 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-06T13:44:20.031 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:19 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:44:21.754 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:21 smithi170 ceph-mon[23272]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-05-06T13:44:21.754 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:21 smithi170 ceph-mon[23272]: mgrmap e27: x(active, since 3m), standbys: y 2024-05-06T13:44:22.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:21 smithi077 ceph-mon[22428]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-05-06T13:44:22.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:21 smithi077 ceph-mon[22428]: mgrmap e27: x(active, since 3m), standbys: y 2024-05-06T13:44:22.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:21 smithi077 ceph-mon[27166]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-05-06T13:44:22.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:21 smithi077 ceph-mon[27166]: mgrmap e27: x(active, since 3m), standbys: y 2024-05-06T13:44:23.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:23 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:23.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:23 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:23.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:23 smithi077 ceph-mon[27166]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-05-06T13:44:23.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:23 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:23.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:23 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:23.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:23 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:44:23.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:23 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:44:23.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:23 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:23.804 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:23 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:23.804 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:23 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:23.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:23 smithi077 ceph-mon[22428]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-05-06T13:44:23.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:23 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:23.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:23 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:23.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:23 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:44:23.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:23 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:44:23.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:23 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:23.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:23 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:23.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:23 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:23.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:23 smithi170 ceph-mon[23272]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-05-06T13:44:23.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:23 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:23.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:23 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:23.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:23 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:44:23.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:23 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:44:23.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:23 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:24.755 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:24 smithi077 ceph-mon[27166]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 655 B/s rd, 0 op/s 2024-05-06T13:44:24.755 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:24 smithi077 ceph-mon[27166]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-06T13:44:24.755 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:24 smithi077 ceph-mon[27166]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:44:24.755 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:24 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:44:24.755 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:24 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:44:24.755 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:24 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:44:24.755 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:24 smithi077 ceph-mon[27166]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:44:24.755 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:24 smithi077 ceph-mon[22428]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 655 B/s rd, 0 op/s 2024-05-06T13:44:24.755 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:24 smithi077 ceph-mon[22428]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-06T13:44:24.755 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:24 smithi077 ceph-mon[22428]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:44:24.756 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:24 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:44:24.756 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:24 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:44:24.756 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:24 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:44:24.756 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:24 smithi077 ceph-mon[22428]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:44:24.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:24 smithi170 ceph-mon[23272]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 655 B/s rd, 0 op/s 2024-05-06T13:44:24.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:24 smithi170 ceph-mon[23272]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-06T13:44:24.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:24 smithi170 ceph-mon[23272]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:44:24.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:24 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:44:24.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:24 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:44:24.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:24 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:44:24.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:24 smithi170 ceph-mon[23272]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:44:27.026 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:26 smithi077 ceph-mon[22428]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 187 B/s rd, 0 op/s 2024-05-06T13:44:27.026 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:26 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:27.026 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:26 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:27.027 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:26 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:44:27.027 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:26 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:44:27.027 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:26 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/761656385' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:44:27.027 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:26 smithi077 ceph-mon[27166]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 187 B/s rd, 0 op/s 2024-05-06T13:44:27.027 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:26 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:27.027 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:26 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:27.027 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:26 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:44:27.027 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:26 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:44:27.027 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:26 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/761656385' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:44:27.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:26 smithi170 ceph-mon[23272]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 187 B/s rd, 0 op/s 2024-05-06T13:44:27.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:26 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:27.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:26 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:27.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:26 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:44:27.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:26 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:44:27.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:26 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/761656385' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:44:27.919 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:27 smithi077 ceph-mon[22428]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:44:27.920 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:27 smithi077 ceph-mon[22428]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:44:27.920 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:27 smithi077 ceph-mon[27166]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:44:27.920 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:27 smithi077 ceph-mon[27166]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:44:28.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:27 smithi170 ceph-mon[23272]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:44:28.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:27 smithi170 ceph-mon[23272]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:44:29.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:28 smithi077 ceph-mon[22428]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 187 B/s rd, 0 op/s 2024-05-06T13:44:29.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:28 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:29.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:28 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:44:29.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:28 smithi077 ceph-mon[27166]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 187 B/s rd, 0 op/s 2024-05-06T13:44:29.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:28 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:29.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:28 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:44:29.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:28 smithi170 ceph-mon[23272]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 187 B/s rd, 0 op/s 2024-05-06T13:44:29.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:28 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:29.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:28 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:44:29.076 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:44:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:44:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:44:29.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:44:29] "GET /metrics HTTP/1.1" 200 37975 "" "Prometheus/2.33.4" 2024-05-06T13:44:31.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:30 smithi077 ceph-mon[22428]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 187 B/s rd, 0 op/s 2024-05-06T13:44:31.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:30 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:31.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:30 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:31.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:30 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:44:31.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:30 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:44:31.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:30 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:31.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:30 smithi077 ceph-mon[27166]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 187 B/s rd, 0 op/s 2024-05-06T13:44:31.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:30 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:31.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:30 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:31.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:30 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:44:31.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:30 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:44:31.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:30 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:31.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:30 smithi170 ceph-mon[23272]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 187 B/s rd, 0 op/s 2024-05-06T13:44:31.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:30 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:31.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:30 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:31.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:30 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:44:31.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:30 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:44:31.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:30 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:32.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:31 smithi077 ceph-mon[22428]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 406 B/s rd, 0 op/s 2024-05-06T13:44:32.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:31 smithi077 ceph-mon[22428]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-05-06T13:44:32.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:31 smithi077 ceph-mon[22428]: Cluster is now healthy 2024-05-06T13:44:32.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:31 smithi077 ceph-mon[27166]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 406 B/s rd, 0 op/s 2024-05-06T13:44:32.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:31 smithi077 ceph-mon[27166]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-05-06T13:44:32.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:31 smithi077 ceph-mon[27166]: Cluster is now healthy 2024-05-06T13:44:32.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:31 smithi170 ceph-mon[23272]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 406 B/s rd, 0 op/s 2024-05-06T13:44:32.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:31 smithi170 ceph-mon[23272]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-05-06T13:44:32.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:31 smithi170 ceph-mon[23272]: Cluster is now healthy 2024-05-06T13:44:34.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:33 smithi077 ceph-mon[22428]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 406 B/s rd, 0 op/s 2024-05-06T13:44:34.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:33 smithi077 ceph-mon[27166]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 406 B/s rd, 0 op/s 2024-05-06T13:44:34.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:33 smithi170 ceph-mon[23272]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 406 B/s rd, 0 op/s 2024-05-06T13:44:36.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:35 smithi077 ceph-mon[22428]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 827 B/s rd, 0 op/s 2024-05-06T13:44:36.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:35 smithi077 ceph-mon[27166]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 827 B/s rd, 0 op/s 2024-05-06T13:44:36.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:35 smithi170 ceph-mon[23272]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 827 B/s rd, 0 op/s 2024-05-06T13:44:38.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:37 smithi077 ceph-mon[27166]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:44:38.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:37 smithi077 ceph-mon[27166]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 827 B/s rd, 0 op/s 2024-05-06T13:44:38.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:37 smithi077 ceph-mon[22428]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:44:38.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:37 smithi077 ceph-mon[22428]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 827 B/s rd, 0 op/s 2024-05-06T13:44:38.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:37 smithi170 ceph-mon[23272]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:44:38.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:37 smithi170 ceph-mon[23272]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 827 B/s rd, 0 op/s 2024-05-06T13:44:38.977 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:44:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:44:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:44:39.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:39 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:44:39] "GET /metrics HTTP/1.1" 200 37975 "" "Prometheus/2.33.4" 2024-05-06T13:44:39.989 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:39 smithi170 ceph-mon[23272]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 827 B/s rd, 0 op/s 2024-05-06T13:44:40.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:39 smithi077 ceph-mon[22428]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 827 B/s rd, 0 op/s 2024-05-06T13:44:40.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:39 smithi077 ceph-mon[27166]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 827 B/s rd, 0 op/s 2024-05-06T13:44:42.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:41 smithi077 ceph-mon[22428]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-06T13:44:42.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:41 smithi077 ceph-mon[27166]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-06T13:44:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:41 smithi170 ceph-mon[23272]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-06T13:44:44.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:43 smithi077 ceph-mon[22428]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:44.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:43 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:44.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:43 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:44:44.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:43 smithi077 ceph-mon[27166]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:44.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:43 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:44.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:43 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:44:44.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:43 smithi170 ceph-mon[23272]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:44.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:43 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:44:44.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:43 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:44:46.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:45 smithi077 ceph-mon[22428]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:44:46.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:45 smithi077 ceph-mon[27166]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:44:46.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:45 smithi170 ceph-mon[23272]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:44:48.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:47 smithi077 ceph-mon[22428]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:44:48.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:47 smithi077 ceph-mon[22428]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:48.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:47 smithi077 ceph-mon[27166]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:44:48.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:47 smithi077 ceph-mon[27166]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:48.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:47 smithi170 ceph-mon[23272]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:44:48.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:47 smithi170 ceph-mon[23272]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:49.076 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:44:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:44:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:44:49.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:49 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:44:49] "GET /metrics HTTP/1.1" 200 37975 "" "Prometheus/2.33.4" 2024-05-06T13:44:50.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:49 smithi077 ceph-mon[27166]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:50.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:49 smithi077 ceph-mon[22428]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:50.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:49 smithi170 ceph-mon[23272]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:52.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:51 smithi077 ceph-mon[22428]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:44:52.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:51 smithi077 ceph-mon[27166]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:44:52.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:51 smithi170 ceph-mon[23272]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:44:54.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:53 smithi077 ceph-mon[22428]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:54.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:53 smithi077 ceph-mon[27166]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:54.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:53 smithi170 ceph-mon[23272]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:56.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:55 smithi077 ceph-mon[22428]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:44:56.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:55 smithi077 ceph-mon[27166]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:44:56.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:55 smithi170 ceph-mon[23272]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:44:57.955 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:57 smithi077 ceph-mon[22428]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:44:57.955 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:57 smithi077 ceph-mon[22428]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:57.956 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:57 smithi077 ceph-mon[27166]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:44:57.956 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:57 smithi077 ceph-mon[27166]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:58.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:57 smithi170 ceph-mon[23272]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:44:58.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:57 smithi170 ceph-mon[23272]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:44:59.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:58 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:44:59.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:58 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:44:59.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:58 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:44:59.076 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:44:58 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:44:58.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:44:59.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:44:59 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:44:59] "GET /metrics HTTP/1.1" 200 37973 "" "Prometheus/2.33.4" 2024-05-06T13:45:00.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:44:59 smithi077 ceph-mon[22428]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:00.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:44:59 smithi077 ceph-mon[27166]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:00.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:44:59 smithi170 ceph-mon[23272]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:02.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:01 smithi077 ceph-mon[22428]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:02.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:01 smithi077 ceph-mon[27166]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:02.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:01 smithi170 ceph-mon[23272]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:04.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:03 smithi077 ceph-mon[22428]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:04.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:03 smithi077 ceph-mon[27166]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:04.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:03 smithi170 ceph-mon[23272]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:06.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:05 smithi077 ceph-mon[22428]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:06.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:05 smithi077 ceph-mon[27166]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:06.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:05 smithi170 ceph-mon[23272]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:08.024 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:07 smithi077 ceph-mon[22428]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:45:08.024 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:07 smithi077 ceph-mon[22428]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:45:08.025 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:07 smithi077 ceph-mon[27166]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:45:08.025 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:07 smithi077 ceph-mon[27166]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:45:08.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:07 smithi170 ceph-mon[23272]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:45:08.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:07 smithi170 ceph-mon[23272]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:45:08.977 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:45:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:45:08.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:45:09.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:45:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:45:09] "GET /metrics HTTP/1.1" 200 37973 "" "Prometheus/2.33.4" 2024-05-06T13:45:10.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:09 smithi077 ceph-mon[22428]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:45:10.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:09 smithi077 ceph-mon[27166]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:45:10.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:09 smithi170 ceph-mon[23272]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:45:12.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:11 smithi077 ceph-mon[22428]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:12.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:11 smithi077 ceph-mon[27166]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:12.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:11 smithi170 ceph-mon[23272]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:14.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:13 smithi077 ceph-mon[22428]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:45:14.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:13 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:45:14.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:13 smithi077 ceph-mon[27166]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:45:14.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:13 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:45:14.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:13 smithi170 ceph-mon[23272]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:45:14.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:13 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:45:16.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:15 smithi077 ceph-mon[22428]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:16.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:15 smithi077 ceph-mon[27166]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:16.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:15 smithi170 ceph-mon[23272]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:18.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:17 smithi077 ceph-mon[22428]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:45:18.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:17 smithi077 ceph-mon[22428]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:18.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:17 smithi077 ceph-mon[27166]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:45:18.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:17 smithi077 ceph-mon[27166]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:18.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:17 smithi170 ceph-mon[23272]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:45:18.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:17 smithi170 ceph-mon[23272]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:19.076 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:45:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:45:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:45:19.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:45:19 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:45:19] "GET /metrics HTTP/1.1" 200 37969 "" "Prometheus/2.33.4" 2024-05-06T13:45:20.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:19 smithi077 ceph-mon[22428]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:20.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:19 smithi077 ceph-mon[27166]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:20.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:19 smithi170 ceph-mon[23272]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:22.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:21 smithi077 ceph-mon[22428]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:22.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:21 smithi077 ceph-mon[27166]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:22.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:21 smithi170 ceph-mon[23272]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:24.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:23 smithi077 ceph-mon[22428]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:24.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:23 smithi077 ceph-mon[27166]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:24.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:23 smithi170 ceph-mon[23272]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:26.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:25 smithi077 ceph-mon[22428]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:26.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:25 smithi077 ceph-mon[27166]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:26.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:25 smithi170 ceph-mon[23272]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:28.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:27 smithi077 ceph-mon[27166]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:45:28.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:27 smithi077 ceph-mon[27166]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:28.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:27 smithi077 ceph-mon[22428]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:45:28.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:27 smithi077 ceph-mon[22428]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:28.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:27 smithi170 ceph-mon[23272]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:45:28.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:27 smithi170 ceph-mon[23272]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:29.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:28 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:45:29.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:28 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:45:29.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:28 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:45:29.076 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:45:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:45:28.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:45:29.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:45:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:45:29] "GET /metrics HTTP/1.1" 200 37974 "" "Prometheus/2.33.4" 2024-05-06T13:45:30.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:29 smithi077 ceph-mon[22428]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:30.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:29 smithi077 ceph-mon[27166]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:30.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:29 smithi170 ceph-mon[23272]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:32.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:31 smithi077 ceph-mon[22428]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:32.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:31 smithi077 ceph-mon[27166]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:32.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:31 smithi170 ceph-mon[23272]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:34.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:33 smithi077 ceph-mon[27166]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:34.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:33 smithi077 ceph-mon[22428]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:34.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:33 smithi170 ceph-mon[23272]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:36.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:36 smithi077 ceph-mon[22428]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:36.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:36 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:45:36.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:36 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:45:36.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:36 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:45:36.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:36 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:45:36.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:36 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:45:36.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:36 smithi077 ceph-mon[27166]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:36.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:36 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:45:36.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:36 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:45:36.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:36 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:45:36.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:36 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:45:36.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:36 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:45:36.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:36 smithi170 ceph-mon[23272]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:36.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:36 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:45:36.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:36 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:45:36.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:36 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:45:36.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:36 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:45:36.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:36 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:45:38.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:38 smithi077 ceph-mon[22428]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:45:38.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:38 smithi077 ceph-mon[22428]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:38.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:38 smithi077 ceph-mon[27166]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:45:38.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:38 smithi077 ceph-mon[27166]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:38.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:38 smithi170 ceph-mon[23272]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:45:38.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:38 smithi170 ceph-mon[23272]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:39.076 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:45:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:45:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:45:39.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:45:39 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:45:39] "GET /metrics HTTP/1.1" 200 37974 "" "Prometheus/2.33.4" 2024-05-06T13:45:40.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:40 smithi077 ceph-mon[22428]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:40.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:40 smithi077 ceph-mon[27166]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:40.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:40 smithi170 ceph-mon[23272]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:42.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:42 smithi077 ceph-mon[22428]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:42.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:42 smithi077 ceph-mon[27166]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:42.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:42 smithi170 ceph-mon[23272]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:44.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:44 smithi077 ceph-mon[22428]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:44.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:44 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:45:44.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:44 smithi077 ceph-mon[27166]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:44.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:44 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:45:44.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:44 smithi170 ceph-mon[23272]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:44.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:44 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:45:46.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:46 smithi077 ceph-mon[22428]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:46.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:46 smithi077 ceph-mon[27166]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:46.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:46 smithi170 ceph-mon[23272]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:48.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:48 smithi077 ceph-mon[22428]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:45:48.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:48 smithi077 ceph-mon[22428]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:48.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:48 smithi077 ceph-mon[27166]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:45:48.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:48 smithi077 ceph-mon[27166]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:48.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:48 smithi170 ceph-mon[23272]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:45:48.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:48 smithi170 ceph-mon[23272]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:49.076 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:45:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:45:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"smithi170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.77\", device_class=\"ssd\", hostname=\"smithi077\", instance=\"172.21.15.170:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.77\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-05-06T13:45:49.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:45:49 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:45:49] "GET /metrics HTTP/1.1" 200 37973 "" "Prometheus/2.33.4" 2024-05-06T13:45:50.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:50 smithi077 ceph-mon[22428]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:50.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:50 smithi077 ceph-mon[27166]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:50.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:50 smithi170 ceph-mon[23272]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:52.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:52 smithi077 ceph-mon[22428]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:52.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:52 smithi077 ceph-mon[27166]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:52.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:52 smithi170 ceph-mon[23272]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:54.305 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:54 smithi077 ceph-mon[22428]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:54.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:54 smithi077 ceph-mon[27166]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:54.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:54 smithi170 ceph-mon[23272]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:56.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:56 smithi077 ceph-mon[22428]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:56.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:56 smithi077 ceph-mon[27166]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:56.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:56 smithi170 ceph-mon[23272]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:45:58.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:58 smithi077 ceph-mon[22428]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:45:58.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:58 smithi077 ceph-mon[22428]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:58.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:58 smithi077 ceph-mon[27166]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:45:58.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:58 smithi077 ceph-mon[27166]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:58.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:58 smithi170 ceph-mon[23272]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:45:58.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:58 smithi170 ceph-mon[23272]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:45:59.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:45:59 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:45:59.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:45:59 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:45:59.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:45:59 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:45:59.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:45:59 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:45:59] "GET /metrics HTTP/1.1" 200 37973 "" "Prometheus/2.33.4" 2024-05-06T13:46:00.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:00 smithi077 ceph-mon[22428]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:00.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:00 smithi077 ceph-mon[27166]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:00.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:00 smithi170 ceph-mon[23272]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:02.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:02 smithi077 ceph-mon[22428]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:02.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:02 smithi077 ceph-mon[27166]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:02.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:02 smithi170 ceph-mon[23272]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:04.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:04 smithi077 ceph-mon[22428]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:04.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:04 smithi077 ceph-mon[27166]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:04.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:04 smithi170 ceph-mon[23272]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:06.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:06 smithi170 ceph-mon[23272]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:06.419 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:06 smithi077 ceph-mon[22428]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:06.419 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:06 smithi077 ceph-mon[27166]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:08.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:08 smithi170 ceph-mon[23272]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:46:08.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:08 smithi170 ceph-mon[23272]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:08.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:08 smithi077 ceph-mon[22428]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:46:08.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:08 smithi077 ceph-mon[22428]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:08.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:08 smithi077 ceph-mon[27166]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:46:08.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:08 smithi077 ceph-mon[27166]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:09.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:46:09] "GET /metrics HTTP/1.1" 200 37973 "" "Prometheus/2.33.4" 2024-05-06T13:46:10.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:10 smithi170 ceph-mon[23272]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:10.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:10 smithi077 ceph-mon[22428]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:10.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:10 smithi077 ceph-mon[27166]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:12.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:12 smithi170 ceph-mon[23272]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:12.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:12 smithi077 ceph-mon[22428]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:12.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:12 smithi077 ceph-mon[27166]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:14.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:14 smithi170 ceph-mon[23272]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:14.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:14 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:46:14.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:14 smithi077 ceph-mon[22428]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:14.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:14 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:46:14.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:14 smithi077 ceph-mon[27166]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:14.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:14 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:46:16.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:16 smithi170 ceph-mon[23272]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:16.421 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:16 smithi077 ceph-mon[22428]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:16.421 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:16 smithi077 ceph-mon[27166]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:18.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:18 smithi170 ceph-mon[23272]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:46:18.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:18 smithi170 ceph-mon[23272]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:18.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:18 smithi077 ceph-mon[22428]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:46:18.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:18 smithi077 ceph-mon[22428]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:18.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:18 smithi077 ceph-mon[27166]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:46:18.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:18 smithi077 ceph-mon[27166]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:19.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:19 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:46:19] "GET /metrics HTTP/1.1" 200 37971 "" "Prometheus/2.33.4" 2024-05-06T13:46:20.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:20 smithi170 ceph-mon[23272]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:20.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:20 smithi077 ceph-mon[22428]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:20.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:20 smithi077 ceph-mon[27166]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:22.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:22 smithi077 ceph-mon[22428]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:22.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:22 smithi077 ceph-mon[27166]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:22.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:22 smithi170 ceph-mon[23272]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:24.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:24 smithi077 ceph-mon[22428]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:24.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:24 smithi077 ceph-mon[27166]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:24.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:24 smithi170 ceph-mon[23272]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:26.354 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:26 smithi077 ceph-mon[22428]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:26.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:26 smithi077 ceph-mon[27166]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:26.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:26 smithi170 ceph-mon[23272]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:28.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:28 smithi077 ceph-mon[22428]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:46:28.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:28 smithi077 ceph-mon[22428]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:28.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:28 smithi077 ceph-mon[27166]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:46:28.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:28 smithi077 ceph-mon[27166]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:28.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:28 smithi170 ceph-mon[23272]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:46:28.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:28 smithi170 ceph-mon[23272]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:29.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:29 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:46:29.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:29 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:46:29.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:29 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:46:29.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:46:29] "GET /metrics HTTP/1.1" 200 37972 "" "Prometheus/2.33.4" 2024-05-06T13:46:30.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:30 smithi077 ceph-mon[22428]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:30.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:30 smithi077 ceph-mon[27166]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:30.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:30 smithi170 ceph-mon[23272]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:31.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:31 smithi077 ceph-mon[22428]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:31.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:31 smithi077 ceph-mon[27166]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:31.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:31 smithi170 ceph-mon[23272]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:33.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:33 smithi170 ceph-mon[23272]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:34.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:33 smithi077 ceph-mon[22428]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:34.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:33 smithi077 ceph-mon[27166]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:36.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:35 smithi077 ceph-mon[22428]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:36.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:35 smithi077 ceph-mon[27166]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:36.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:35 smithi170 ceph-mon[23272]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:36.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:36 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:46:36.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:36 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:46:36.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:36 smithi077 ceph-mon[22428]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:46:36.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:36 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:46:36.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:36 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:46:36.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:36 smithi077 ceph-mon[27166]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:46:37.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:36 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:46:37.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:36 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:46:37.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:36 smithi170 ceph-mon[23272]: from='mgr.34775 ' entity='mgr.x' 2024-05-06T13:46:38.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:37 smithi077 ceph-mon[22428]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:46:38.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:37 smithi077 ceph-mon[22428]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:38.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:37 smithi077 ceph-mon[27166]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:46:38.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:37 smithi077 ceph-mon[27166]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:38.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:37 smithi170 ceph-mon[23272]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:46:38.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:37 smithi170 ceph-mon[23272]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:39.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:39 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:46:39] "GET /metrics HTTP/1.1" 200 37972 "" "Prometheus/2.33.4" 2024-05-06T13:46:40.027 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:39 smithi077 ceph-mon[22428]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:40.027 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:39 smithi077 ceph-mon[27166]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:40.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:39 smithi170 ceph-mon[23272]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:40.389 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch ps' 2024-05-06T13:46:41.989 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:41 smithi077 ceph-mon[22428]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:41.989 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:41 smithi077 ceph-mon[27166]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:41 smithi170 ceph-mon[23272]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:42.423 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T13:46:42.423 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (5m) 67s ago 13m 23.8M - 0.23.0 ba2b418f427c 0f855548777f 2024-05-06T13:46:42.423 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (5m) 2m ago 13m 53.5M - 8.3.5 dad864ee21e9 fdcecac79925 2024-05-06T13:46:42.423 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (2m) 67s ago 12m 44.3M - 3.5 e1d6a67b021e b32d96cb5d0d 2024-05-06T13:46:42.423 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283 running (9m) 2m ago 18m 513M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 63ef6825ff68 2024-05-06T13:46:42.423 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (2m) 67s ago 19m 451M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T13:46:42.423 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (19m) 67s ago 20m 56.4M 2048M 17.2.0 e1d6a67b021e ac46f91694f8 2024-05-06T13:46:42.423 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (18m) 2m ago 18m 49.9M 2048M 17.2.0 e1d6a67b021e 24be6fce9d28 2024-05-06T13:46:42.423 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (18m) 67s ago 18m 43.6M 2048M 17.2.0 e1d6a67b021e 83af336ef762 2024-05-06T13:46:42.423 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (6m) 67s ago 14m 17.2M - 1.3.1 1dbe0e931976 54aee5b22b1c 2024-05-06T13:46:42.423 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (5m) 2m ago 14m 19.5M - 1.3.1 1dbe0e931976 ca5df43a7469 2024-05-06T13:46:42.424 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (18m) 67s ago 18m 52.5M 2992M 17.2.0 e1d6a67b021e 2ccee2bb6e96 2024-05-06T13:46:42.424 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (17m) 67s ago 17m 56.6M 2992M 17.2.0 e1d6a67b021e 1d9e6609b585 2024-05-06T13:46:42.424 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (17m) 67s ago 17m 51.6M 2992M 17.2.0 e1d6a67b021e 37a61d295111 2024-05-06T13:46:42.424 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (16m) 67s ago 16m 50.9M 2992M 17.2.0 e1d6a67b021e 8feb7c077aaa 2024-05-06T13:46:42.424 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (16m) 2m ago 16m 54.4M 3248M 17.2.0 e1d6a67b021e f865acc44a06 2024-05-06T13:46:42.424 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (16m) 2m ago 16m 50.9M 3248M 17.2.0 e1d6a67b021e a55bd4122f6f 2024-05-06T13:46:42.424 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (15m) 2m ago 15m 49.5M 3248M 17.2.0 e1d6a67b021e 362938a0b357 2024-05-06T13:46:42.424 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (15m) 2m ago 15m 51.9M 3248M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T13:46:42.424 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (5m) 2m ago 14m 69.3M - 2.33.4 514e6a882f6e c45c7b7e4059 2024-05-06T13:46:42.424 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (13m) 67s ago 13m 74.7M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T13:46:42.424 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (13m) 2m ago 13m 72.5M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T13:46:43.042 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph versions' 2024-05-06T13:46:44.024 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:43 smithi077 ceph-mon[22428]: from='client.25177 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:46:44.024 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:43 smithi077 ceph-mon[22428]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:44.024 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:43 smithi077 ceph-mon[22428]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:46:44.024 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:43 smithi077 ceph-mon[27166]: from='client.25177 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:46:44.025 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:43 smithi077 ceph-mon[27166]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:44.025 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:43 smithi077 ceph-mon[27166]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:46:44.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:43 smithi170 ceph-mon[23272]: from='client.25177 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:46:44.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:43 smithi170 ceph-mon[23272]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:44.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:43 smithi170 ceph-mon[23272]: from='mgr.34775 172.21.15.170:0/3110405897' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:46:45.079 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:46:45.080 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-05-06T13:46:45.080 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-05-06T13:46:45.080 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:46:45.080 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-05-06T13:46:45.080 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T13:46:45.080 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:46:45.080 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-05-06T13:46:45.080 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-06T13:46:45.080 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:46:45.080 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": {}, 2024-05-06T13:46:45.080 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw": { 2024-05-06T13:46:45.080 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T13:46:45.080 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:46:45.081 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-05-06T13:46:45.081 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-05-06T13:46:45.081 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T13:46:45.081 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T13:46:45.081 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:46:45.647 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph health detail' 2024-05-06T13:46:45.952 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:45 smithi077 ceph-mon[22428]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:45.952 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:45 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2277539105' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:46:45.952 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:45 smithi077 ceph-mon[27166]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:45.952 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:45 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2277539105' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:46:46.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:45 smithi170 ceph-mon[23272]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:46.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:45 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2277539105' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:46:47.698 INFO:teuthology.orchestra.run.smithi077.stdout:HEALTH_OK 2024-05-06T13:46:48.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:47 smithi077 ceph-mon[22428]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:46:48.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:47 smithi077 ceph-mon[22428]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:48.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:47 smithi077 ceph-mon[27166]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:46:48.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:47 smithi077 ceph-mon[27166]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:48.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:47 smithi170 ceph-mon[23272]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:46:48.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:47 smithi170 ceph-mon[23272]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:48.327 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph -s' 2024-05-06T13:46:48.975 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:48 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2036199631' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T13:46:48.975 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:48 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2036199631' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T13:46:49.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:48 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2036199631' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T13:46:49.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:49 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:46:49] "GET /metrics HTTP/1.1" 200 37973 "" "Prometheus/2.33.4" 2024-05-06T13:46:49.842 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:49 smithi077 ceph-mon[22428]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:49.842 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:49 smithi077 ceph-mon[27166]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:50.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:49 smithi170 ceph-mon[23272]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:50.331 INFO:teuthology.orchestra.run.smithi077.stdout: cluster: 2024-05-06T13:46:50.331 INFO:teuthology.orchestra.run.smithi077.stdout: id: 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:46:50.331 INFO:teuthology.orchestra.run.smithi077.stdout: health: HEALTH_OK 2024-05-06T13:46:50.332 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:46:50.332 INFO:teuthology.orchestra.run.smithi077.stdout: services: 2024-05-06T13:46:50.332 INFO:teuthology.orchestra.run.smithi077.stdout: mon: 3 daemons, quorum a,c,b (age 18m) 2024-05-06T13:46:50.332 INFO:teuthology.orchestra.run.smithi077.stdout: mgr: x(active, since 6m), standbys: y 2024-05-06T13:46:50.332 INFO:teuthology.orchestra.run.smithi077.stdout: osd: 8 osds: 8 up (since 15m), 8 in (since 15m) 2024-05-06T13:46:50.332 INFO:teuthology.orchestra.run.smithi077.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-05-06T13:46:50.332 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:46:50.333 INFO:teuthology.orchestra.run.smithi077.stdout: data: 2024-05-06T13:46:50.333 INFO:teuthology.orchestra.run.smithi077.stdout: pools: 6 pools, 161 pgs 2024-05-06T13:46:50.333 INFO:teuthology.orchestra.run.smithi077.stdout: objects: 209 objects, 585 KiB 2024-05-06T13:46:50.333 INFO:teuthology.orchestra.run.smithi077.stdout: usage: 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:46:50.333 INFO:teuthology.orchestra.run.smithi077.stdout: pgs: 161 active+clean 2024-05-06T13:46:50.333 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:46:50.333 INFO:teuthology.orchestra.run.smithi077.stdout: io: 2024-05-06T13:46:50.334 INFO:teuthology.orchestra.run.smithi077.stdout: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-05-06T13:46:50.334 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:46:50.969 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:50 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/1095030753' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-06T13:46:50.969 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:50 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/1095030753' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-06T13:46:50.994 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph mgr fail' 2024-05-06T13:46:51.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:50 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/1095030753' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-06T13:46:52.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:51 smithi077 ceph-mon[22428]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:52.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:51 smithi077 ceph-mon[27166]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:52.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:51 smithi170 ceph-mon[23272]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:46:53.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:53 smithi077 ceph-mon[22428]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:53.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:53 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/887196663' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-05-06T13:46:53.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:53 smithi077 ceph-mon[22428]: osdmap e104: 8 total, 8 up, 8 in 2024-05-06T13:46:53.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:53 smithi077 ceph-mon[27166]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:53.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:53 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/887196663' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-05-06T13:46:53.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:53 smithi077 ceph-mon[27166]: osdmap e104: 8 total, 8 up, 8 in 2024-05-06T13:46:53.803 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:46:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:46:53] ENGINE Bus STOPPING 2024-05-06T13:46:53.803 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:46:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:46:53] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-06T13:46:53.803 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:46:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:46:53] ENGINE Bus STOPPED 2024-05-06T13:46:54.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:53 smithi170 ceph-mon[23272]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:46:54.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:53 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/887196663' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-05-06T13:46:54.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:53 smithi170 ceph-mon[23272]: osdmap e104: 8 total, 8 up, 8 in 2024-05-06T13:46:54.076 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:53 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:46:53.728+0000 7f899e209700 -1 mgr handle_mgr_map I was active but no longer am 2024-05-06T13:46:54.077 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:53 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ignoring --setuser ceph since I am not root 2024-05-06T13:46:54.077 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:53 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ignoring --setgroup ceph since I am not root 2024-05-06T13:46:54.077 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:53 smithi170 ceph-mgr[43056]: -- 172.21.15.170:0/1519262256 <== mon.2 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561d229fe5a0 con 0x561d229d2800 2024-05-06T13:46:54.077 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:53 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:46:53.903+0000 7f570ef25200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T13:46:54.151 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:46:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:46:53] ENGINE Bus STARTING 2024-05-06T13:46:54.151 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:46:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:46:53] ENGINE Serving on http://:::9283 2024-05-06T13:46:54.152 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:46:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:46:53] ENGINE Bus STARTED 2024-05-06T13:46:54.395 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'sleep 180' 2024-05-06T13:46:54.531 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:54 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:46:54.093+0000 7f570ef25200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:46:54.532 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:54 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:46:54.202+0000 7f570ef25200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T13:46:54.532 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:54 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:46:54.530+0000 7f570ef25200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T13:46:54.826 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:54 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:46:54.626+0000 7f570ef25200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T13:46:54.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:54 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/887196663' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-05-06T13:46:54.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:54 smithi170 ceph-mon[23272]: mgrmap e28: y(active, starting, since 0.557022s) 2024-05-06T13:46:54.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:54 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:46:54.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:54 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:46:54.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:54 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:46:54.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:54 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T13:46:54.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:54 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:46:54.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:54 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:46:54.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:54 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:46:54.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:54 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:46:54.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:54 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:46:54.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:54 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:46:54.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:54 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:46:54.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:54 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:46:54.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:54 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T13:46:54.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:54 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T13:46:54.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:54 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T13:46:54.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:54 smithi170 ceph-mon[23272]: Manager daemon y is now available 2024-05-06T13:46:54.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:54 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:46:54.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:54 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:46:54.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:54 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:46:55.045 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/887196663' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-05-06T13:46:55.045 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[22428]: mgrmap e28: y(active, starting, since 0.557022s) 2024-05-06T13:46:55.046 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:46:55.046 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:46:55.046 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:46:55.046 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T13:46:55.046 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:46:55.046 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:46:55.046 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:46:55.046 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:46:55.046 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:46:55.046 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:46:55.046 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:46:55.046 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:46:55.046 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T13:46:55.046 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T13:46:55.047 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T13:46:55.047 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[22428]: Manager daemon y is now available 2024-05-06T13:46:55.047 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:46:55.047 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:46:55.047 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:46:55.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/887196663' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-05-06T13:46:55.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[27166]: mgrmap e28: y(active, starting, since 0.557022s) 2024-05-06T13:46:55.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:46:55.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:46:55.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:46:55.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T13:46:55.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:46:55.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:46:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:46:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:46:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:46:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:46:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:46:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:46:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T13:46:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T13:46:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T13:46:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[27166]: Manager daemon y is now available 2024-05-06T13:46:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:46:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:46:55.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:54 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:46:55.876 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:55 smithi170 ceph-mon[23272]: mgrmap e29: y(active, since 1.56813s) 2024-05-06T13:46:55.876 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:55 smithi170 ceph-mon[23272]: [06/May/2024:13:46:55] ENGINE Bus STARTING 2024-05-06T13:46:55.876 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:55 smithi170 ceph-mon[23272]: [06/May/2024:13:46:55] ENGINE Serving on https://172.21.15.77:7150 2024-05-06T13:46:55.877 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:46:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:46:55.676Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.170:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.170:8765: connect: connection refused" 2024-05-06T13:46:55.877 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:46:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:46:55.676Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.170:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.170:8765: connect: connection refused" 2024-05-06T13:46:55.877 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:46:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:46:55.677Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.170:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.170:8765: connect: connection refused" 2024-05-06T13:46:55.877 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:46:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:46:55.677Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.170:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.170:8765: connect: connection refused" 2024-05-06T13:46:55.877 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:46:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:46:55.678Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.170:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.170:8765: connect: connection refused" 2024-05-06T13:46:55.950 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:55 smithi077 ceph-mon[22428]: mgrmap e29: y(active, since 1.56813s) 2024-05-06T13:46:55.950 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:55 smithi077 ceph-mon[22428]: [06/May/2024:13:46:55] ENGINE Bus STARTING 2024-05-06T13:46:55.950 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:55 smithi077 ceph-mon[22428]: [06/May/2024:13:46:55] ENGINE Serving on https://172.21.15.77:7150 2024-05-06T13:46:55.950 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:55 smithi077 ceph-mon[27166]: mgrmap e29: y(active, since 1.56813s) 2024-05-06T13:46:55.950 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:55 smithi077 ceph-mon[27166]: [06/May/2024:13:46:55] ENGINE Bus STARTING 2024-05-06T13:46:55.951 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:55 smithi077 ceph-mon[27166]: [06/May/2024:13:46:55] ENGINE Serving on https://172.21.15.77:7150 2024-05-06T13:46:56.241 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:46:55.875+0000 7f570ef25200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T13:46:56.241 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:56 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:46:56.050+0000 7f570ef25200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T13:46:56.241 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:56 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:46:56.239+0000 7f570ef25200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T13:46:56.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:56 smithi170 ceph-mon[23272]: [06/May/2024:13:46:55] ENGINE Serving on http://172.21.15.77:8765 2024-05-06T13:46:56.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:56 smithi170 ceph-mon[23272]: [06/May/2024:13:46:55] ENGINE Bus STARTED 2024-05-06T13:46:56.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:56 smithi170 ceph-mon[23272]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:46:56.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:56 smithi170 ceph-mon[23272]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:46:56.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:56 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:46:56.556+0000 7f570ef25200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T13:46:56.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:56 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:46:56.660+0000 7f570ef25200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T13:46:57.028 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:56 smithi077 ceph-mon[22428]: [06/May/2024:13:46:55] ENGINE Serving on http://172.21.15.77:8765 2024-05-06T13:46:57.029 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:56 smithi077 ceph-mon[22428]: [06/May/2024:13:46:55] ENGINE Bus STARTED 2024-05-06T13:46:57.029 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:56 smithi077 ceph-mon[22428]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:46:57.029 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:56 smithi077 ceph-mon[22428]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:46:57.029 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:56 smithi077 ceph-mon[27166]: [06/May/2024:13:46:55] ENGINE Serving on http://172.21.15.77:8765 2024-05-06T13:46:57.029 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:56 smithi077 ceph-mon[27166]: [06/May/2024:13:46:55] ENGINE Bus STARTED 2024-05-06T13:46:57.029 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:56 smithi077 ceph-mon[27166]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:46:57.029 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:56 smithi077 ceph-mon[27166]: from='client.34973 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:46:57.319 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:56 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:46:56.869+0000 7f570ef25200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T13:46:57.319 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:46:57.066+0000 7f570ef25200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:46:58.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:57 smithi077 ceph-mon[22428]: mgrmap e30: y(active, since 3s) 2024-05-06T13:46:58.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:57 smithi077 ceph-mon[27166]: mgrmap e30: y(active, since 3s) 2024-05-06T13:46:58.057 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:57 smithi170 ceph-mon[23272]: mgrmap e30: y(active, since 3s) 2024-05-06T13:46:58.057 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:46:57.695+0000 7f570ef25200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T13:46:58.307 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:58 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:46:58.147+0000 7f570ef25200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T13:46:58.307 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:58 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:46:58.247+0000 7f570ef25200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T13:46:58.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:58 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:46:58.455+0000 7f570ef25200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-06T13:46:59.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:59 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:46:59.020+0000 7f570ef25200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T13:46:59.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:59 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:46:59.120+0000 7f570ef25200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T13:46:59.752 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:59 smithi170 ceph-mon[23272]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:46:59.753 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:59 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:46:59.753 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:59 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:46:59.753 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:46:59 smithi170 ceph-mon[23272]: mgrmap e31: y(active, since 5s) 2024-05-06T13:46:59.753 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:59 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:46:59.386+0000 7f570ef25200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T13:46:59.753 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:59 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:46:59.527+0000 7f570ef25200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T13:46:59.753 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:59 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:46:59.628+0000 7f570ef25200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T13:46:59.753 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:59 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:46:59.752+0000 7f570ef25200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T13:46:59.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:59 smithi077 ceph-mon[27166]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:46:59.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:59 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:46:59.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:59 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:46:59.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:46:59 smithi077 ceph-mon[27166]: mgrmap e31: y(active, since 5s) 2024-05-06T13:46:59.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:59 smithi077 ceph-mon[22428]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:46:59.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:59 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:46:59.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:59 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:46:59.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:46:59 smithi077 ceph-mon[22428]: mgrmap e31: y(active, since 5s) 2024-05-06T13:47:00.076 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:46:59 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:46:59.848+0000 7f570ef25200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T13:47:01.000 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:00 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:01.000 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:00 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:01.000 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:00 smithi170 ceph-mon[23272]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:47:01.000 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:00 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:01.000 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:00 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:01.000 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:47:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:47:00.672+0000 7f570ef25200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T13:47:01.000 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:47:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:47:00.891+0000 7f570ef25200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T13:47:01.001 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:47:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:47:00] ENGINE Bus STARTING 2024-05-06T13:47:01.001 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:47:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: CherryPy Checker: 2024-05-06T13:47:01.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:00 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:01.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:00 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:01.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:00 smithi077 ceph-mon[22428]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:47:01.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:00 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:01.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:00 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:01.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:00 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:01.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:00 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:01.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:00 smithi077 ceph-mon[27166]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:47:01.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:00 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:01.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:00 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:01.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:47:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: The Application mounted at '' has an empty config. 2024-05-06T13:47:01.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:47:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:47:01.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:47:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:47:01] ENGINE Serving on http://:::9283 2024-05-06T13:47:01.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:47:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:47:01] ENGINE Bus STARTED 2024-05-06T13:47:02.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[22428]: Standby manager daemon x started 2024-05-06T13:47:02.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T13:47:02.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:47:02.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T13:47:02.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:47:02.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:02.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:02.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:47:02.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:47:02.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[22428]: Updating smithi077:/etc/ceph/ceph.conf 2024-05-06T13:47:02.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[22428]: Updating smithi170:/etc/ceph/ceph.conf 2024-05-06T13:47:02.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[22428]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:47:02.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[22428]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:47:02.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[22428]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:47:02.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[22428]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:47:02.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[27166]: Standby manager daemon x started 2024-05-06T13:47:02.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T13:47:02.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:47:02.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T13:47:02.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:47:02.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:02.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:02.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:47:02.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:47:02.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[27166]: Updating smithi077:/etc/ceph/ceph.conf 2024-05-06T13:47:02.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[27166]: Updating smithi170:/etc/ceph/ceph.conf 2024-05-06T13:47:02.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[27166]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:47:02.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[27166]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:47:02.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[27166]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:47:02.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:01 smithi077 ceph-mon[27166]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:47:02.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:01 smithi170 ceph-mon[23272]: Standby manager daemon x started 2024-05-06T13:47:02.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:01 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T13:47:02.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:01 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:47:02.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:01 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T13:47:02.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:01 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:47:02.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:01 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:02.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:01 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:02.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:01 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:47:02.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:01 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:47:02.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:01 smithi170 ceph-mon[23272]: Updating smithi077:/etc/ceph/ceph.conf 2024-05-06T13:47:02.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:01 smithi170 ceph-mon[23272]: Updating smithi170:/etc/ceph/ceph.conf 2024-05-06T13:47:02.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:01 smithi170 ceph-mon[23272]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:47:02.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:01 smithi170 ceph-mon[23272]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:47:02.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:01 smithi170 ceph-mon[23272]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:47:02.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:01 smithi170 ceph-mon[23272]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:47:03.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[27166]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-05-06T13:47:03.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[27166]: mgrmap e32: y(active, since 8s), standbys: x 2024-05-06T13:47:03.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T13:47:03.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[27166]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:47:03.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[27166]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:47:03.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:03.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:03.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:03.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:03.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:03.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[27166]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:47:03.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:47:03.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:47:03.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[27166]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:47:03.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[22428]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-05-06T13:47:03.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[22428]: mgrmap e32: y(active, since 8s), standbys: x 2024-05-06T13:47:03.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T13:47:03.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[22428]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:47:03.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[22428]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:47:03.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:03.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:03.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:03.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:03.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:03.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[22428]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:47:03.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:47:03.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:47:03.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:02 smithi077 ceph-mon[22428]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:47:03.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:02 smithi170 ceph-mon[23272]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-05-06T13:47:03.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:02 smithi170 ceph-mon[23272]: mgrmap e32: y(active, since 8s), standbys: x 2024-05-06T13:47:03.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:02 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T13:47:03.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:02 smithi170 ceph-mon[23272]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:47:03.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:02 smithi170 ceph-mon[23272]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:47:03.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:02 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:03.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:02 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:03.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:02 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:03.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:02 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:03.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:02 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:03.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:02 smithi170 ceph-mon[23272]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:47:03.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:02 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:47:03.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:02 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:47:03.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:02 smithi170 ceph-mon[23272]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:47:05.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:04 smithi077 ceph-mon[22428]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-06T13:47:05.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:04 smithi077 ceph-mon[27166]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-06T13:47:05.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:04 smithi170 ceph-mon[23272]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-06T13:47:05.746 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:47:05.447+0000 7f5d3476d700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:47:05.746 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Creating ceph-iscsi config... 2024-05-06T13:47:05.747 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:47:05.747 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:47:05.747 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:47:05.747 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:47:05.747 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:47:05.747 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Traceback (most recent call last): 2024-05-06T13:47:05.747 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:47:05.747 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: return _run_code(code, main_globals, None, 2024-05-06T13:47:05.747 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:47:05.747 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: exec(code, run_globals) 2024-05-06T13:47:05.747 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:47:05.747 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:47:05.747 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:47:05.747 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:47:05.747 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:47:05.748 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:47:05.748 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:47:05.748 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:47:05.748 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:47:05.748 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Traceback (most recent call last): 2024-05-06T13:47:05.748 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:47:05.748 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:47:05.748 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:47:05.749 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:47:05.749 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:47:05.749 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:47:05.749 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:47:05.749 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: return future.result(timeout) 2024-05-06T13:47:05.749 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:47:05.749 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: return self.__get_result() 2024-05-06T13:47:05.749 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:47:05.749 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: raise self._exception 2024-05-06T13:47:05.749 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:47:05.749 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: use_current_daemon_image=reconfig, 2024-05-06T13:47:05.749 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:47:05.749 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:47:05.749 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:47:05.749 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Creating ceph-iscsi config... 2024-05-06T13:47:05.750 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:47:05.750 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:47:05.750 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:47:05.750 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:47:05.750 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:47:05.750 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Traceback (most recent call last): 2024-05-06T13:47:05.750 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:47:05.750 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: return _run_code(code, main_globals, None, 2024-05-06T13:47:05.750 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:47:05.750 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: exec(code, run_globals) 2024-05-06T13:47:05.750 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:47:05.750 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:47:05.750 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:47:05.750 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:47:05.751 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:47:05.751 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:47:05.751 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:47:05.751 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:47:05.751 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:47:06.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:05 smithi077 ceph-mon[22428]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:47:06.052 INFO:journalctl@ceph.mon.a.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:47:06.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:47:06.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:47:06.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:47:06.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:47:06.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:47:06.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:47:06.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:47:06.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:47:06.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:47:06.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:47:06.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:47:06.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:47:06.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:47:06.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:47:06.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:47:06.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:47:06.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:47:06.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:47:06.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:47:06.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:47:06.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:47:06.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:47:06.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:47:06.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:47:06.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:47:06.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:47:06.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:47:06.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: return future.result(timeout) 2024-05-06T13:47:06.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:47:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: return self.__get_result() 2024-05-06T13:47:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:47:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: raise self._exception 2024-05-06T13:47:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:47:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: use_current_daemon_image=reconfig, 2024-05-06T13:47:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:47:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:47:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:47:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:47:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:47:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:47:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:47:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:47:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:47:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:47:06.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:47:06.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:47:06.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:47:06.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:47:06.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:47:06.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:47:06.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:47:06.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:47:06.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:05 smithi077 ceph-mon[27166]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:47:06.056 INFO:journalctl@ceph.mon.c.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:47:06.056 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:47:06.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:47:06.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:47:06.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:47:06.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:47:06.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:47:06.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:47:06.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:47:06.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:47:06.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:47:06.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:47:06.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:47:06.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:47:06.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:47:06.057 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:47:06.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:47:06.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:47:06.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:47:06.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:47:06.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:47:06.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:47:06.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:47:06.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:47:06.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:47:06.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:47:06.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:47:06.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:47:06.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: return future.result(timeout) 2024-05-06T13:47:06.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:47:06.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: return self.__get_result() 2024-05-06T13:47:06.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:47:06.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: raise self._exception 2024-05-06T13:47:06.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:47:06.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: use_current_daemon_image=reconfig, 2024-05-06T13:47:06.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:47:06.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:47:06.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:47:06.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:47:06.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:47:06.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:47:06.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:47:06.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:47:06.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:47:06.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:47:06.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:47:06.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:47:06.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:47:06.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:47:06.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:47:06.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:47:06.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:47:06.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:47:06.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:47:06.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:47:06.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:47:06.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:47:06.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:47:06.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:05 smithi077 ceph-mon[27166]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-06T13:47:06.061 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:47:06.061 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:47:06.061 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:47:06.061 INFO:journalctl@ceph.mon.a.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:47:06.061 INFO:journalctl@ceph.mon.a.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:47:06.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:05 smithi077 ceph-mon[22428]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-06T13:47:06.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:05 smithi170 ceph-mon[23272]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:47:06.076 INFO:journalctl@ceph.mon.b.smithi170.stdout: Creating ceph-iscsi config... 2024-05-06T13:47:06.076 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:47:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:47:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:47:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:47:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:47:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: Traceback (most recent call last): 2024-05-06T13:47:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:47:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:47:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:47:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: exec(code, run_globals) 2024-05-06T13:47:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:47:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:47:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:47:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:47:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:47:06.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:47:06.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:47:06.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:47:06.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:47:06.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: Traceback (most recent call last): 2024-05-06T13:47:06.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:47:06.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:47:06.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:47:06.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:47:06.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:47:06.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:47:06.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:47:06.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: return future.result(timeout) 2024-05-06T13:47:06.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:47:06.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: return self.__get_result() 2024-05-06T13:47:06.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:47:06.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: raise self._exception 2024-05-06T13:47:06.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:47:06.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: use_current_daemon_image=reconfig, 2024-05-06T13:47:06.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:47:06.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:47:06.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:47:06.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: Creating ceph-iscsi config... 2024-05-06T13:47:06.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:47:06.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:47:06.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:47:06.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:47:06.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:47:06.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: Traceback (most recent call last): 2024-05-06T13:47:06.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:47:06.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:47:06.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:47:06.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: exec(code, run_globals) 2024-05-06T13:47:06.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:47:06.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:47:06.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:47:06.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:47:06.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:47:06.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:47:06.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:47:06.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:47:06.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:47:06.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:05 smithi170 ceph-mon[23272]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-06T13:47:07.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:06 smithi077 ceph-mon[22428]: Reconfiguring daemon prometheus.a on smithi170 2024-05-06T13:47:07.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:06 smithi077 ceph-mon[22428]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-06T13:47:07.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:06 smithi077 ceph-mon[27166]: Reconfiguring daemon prometheus.a on smithi170 2024-05-06T13:47:07.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:06 smithi077 ceph-mon[27166]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-06T13:47:07.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:06 smithi170 ceph-mon[23272]: Reconfiguring daemon prometheus.a on smithi170 2024-05-06T13:47:07.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:06 smithi170 ceph-mon[23272]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-06T13:47:08.887 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:08 smithi170 ceph-mon[23272]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:47:09.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:08 smithi077 ceph-mon[22428]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:47:09.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:08 smithi077 ceph-mon[27166]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:47:09.169 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:09 smithi170 systemd[1]: Stopping Ceph prometheus.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:47:09.169 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:09 smithi170 bash[52114]: Error: no container with name or ID "ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus.a" found: no such container 2024-05-06T13:47:09.442 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:47:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:47:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-05-06T13:47:09.442 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:47:09.200Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-05-06T13:47:09.442 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:47:09.200Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-05-06T13:47:09.442 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:47:09.200Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-05-06T13:47:09.443 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:47:09.200Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-05-06T13:47:09.443 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:47:09.200Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-05-06T13:47:09.443 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:47:09.200Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-05-06T13:47:09.443 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:47:09.200Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-05-06T13:47:09.443 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:47:09.200Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-05-06T13:47:09.443 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:47:09.200Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-05-06T13:47:09.443 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:47:09.200Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-05-06T13:47:09.443 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:47:09.200Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-05-06T13:47:09.443 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[46650]: ts=2024-05-06T13:47:09.200Z caller=main.go:1066 level=info msg="See you next time!" 2024-05-06T13:47:09.443 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:09 smithi170 podman[52125]: 2024-05-06 13:47:09.210641594 +0000 UTC m=+0.133935156 container died c45c7b7e4059b598cd2b8032fd93d947a2d03a4c4e479b0ea38bd6ecf2a9fec8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:47:09.751 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:09 smithi170 podman[52125]: 2024-05-06 13:47:09.558874267 +0000 UTC m=+0.482167825 container cleanup c45c7b7e4059b598cd2b8032fd93d947a2d03a4c4e479b0ea38bd6ecf2a9fec8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:47:09.751 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:09 smithi170 bash[52125]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a 2024-05-06T13:47:09.751 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:09 smithi170 bash[52151]: Error: no container with name or ID "ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus.a" found: no such container 2024-05-06T13:47:10.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:09 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:47:10.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:09 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:47:10.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:09 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:47:10.076 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:09 smithi170 podman[52139]: 2024-05-06 13:47:09.860213689 +0000 UTC m=+0.648637606 container remove c45c7b7e4059b598cd2b8032fd93d947a2d03a4c4e479b0ea38bd6ecf2a9fec8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:47:10.077 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:09 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-05-06T13:47:10.077 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:09 smithi170 systemd[1]: Stopped Ceph prometheus.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:47:10.077 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:09 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@prometheus.a.service: Consumed 1.865s CPU time. 2024-05-06T13:47:10.077 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:09 smithi170 systemd[1]: Starting Ceph prometheus.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:47:10.388 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 podman[52221]: 2024-05-06 13:47:09.997707861 +0000 UTC m=+0.018855801 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-05-06T13:47:10.389 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 podman[52221]: 2024-05-06 13:47:10.136528622 +0000 UTC m=+0.157676552 container create 7550480b5a53088a870bfe3f3d9312fd1c1127babe29f46a60b6592852dfe332 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:47:10.389 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 podman[52221]: 2024-05-06 13:47:10.388802666 +0000 UTC m=+0.409950597 container init 7550480b5a53088a870bfe3f3d9312fd1c1127babe29f46a60b6592852dfe332 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:47:10.677 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 podman[52221]: 2024-05-06 13:47:10.391738633 +0000 UTC m=+0.412886569 container start 7550480b5a53088a870bfe3f3d9312fd1c1127babe29f46a60b6592852dfe332 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:47:10.677 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:47:10.416Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-05-06T13:47:10.677 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:47:10.416Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-05-06T13:47:10.677 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:47:10.416Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-05-06T13:47:10.677 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:47:10.416Z caller=main.go:518 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi170 (none))" 2024-05-06T13:47:10.677 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:47:10.416Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-05-06T13:47:10.678 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:47:10.416Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-05-06T13:47:10.678 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:47:10.417Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-05-06T13:47:10.678 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:47:10.418Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-05-06T13:47:10.678 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:47:10.419Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-05-06T13:47:10.678 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:47:10.421Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-05-06T13:47:10.678 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:47:10.421Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.308µs 2024-05-06T13:47:10.678 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:47:10.421Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-05-06T13:47:10.678 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 bash[52221]: 7550480b5a53088a870bfe3f3d9312fd1c1127babe29f46a60b6592852dfe332 2024-05-06T13:47:10.678 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 systemd[1]: Started Ceph prometheus.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:47:10.678 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:47:10.632Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-05-06T13:47:10.941 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:47:10.886Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-05-06T13:47:10.942 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:47:10.941Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-05-06T13:47:10.942 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:47:10.942Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-05-06T13:47:10.942 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:47:10.942Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=31.601µs wal_replay_duration=520.511932ms total_replay_duration=520.55758ms 2024-05-06T13:47:11.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:10 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:47:10] ENGINE Bus STOPPING 2024-05-06T13:47:11.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:10 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:47:10] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-06T13:47:11.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:10 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:47:10] ENGINE Bus STOPPED 2024-05-06T13:47:11.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:10 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:47:10] ENGINE Bus STARTING 2024-05-06T13:47:11.053 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:10 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:47:10] ENGINE Serving on http://:::9283 2024-05-06T13:47:11.053 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:10 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:47:10] ENGINE Bus STARTED 2024-05-06T13:47:11.326 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:47:10.945Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-05-06T13:47:11.326 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:47:10.945Z caller=main.go:947 level=info msg="TSDB started" 2024-05-06T13:47:11.326 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:47:10.945Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-05-06T13:47:11.326 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:47:10.974Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=29.295713ms db_storage=841ns remote_storage=1.471µs web_handler=492ns query_engine=745ns scrape=757.639µs scrape_sd=194.569µs notify=17.218µs notify_sd=15.547µs rules=28.008998ms 2024-05-06T13:47:11.326 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:47:10 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:47:10.974Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-05-06T13:47:11.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:11 smithi077 ceph-mon[27166]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:47:11.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:11 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:11.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:11 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:11.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:11 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:47:11.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:11 smithi077 ceph-mon[27166]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:47:11.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:11 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi170.front.sepia.ceph.com:9095"}]: dispatch 2024-05-06T13:47:11.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:11 smithi077 ceph-mon[27166]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi170.front.sepia.ceph.com:9095"}]: dispatch 2024-05-06T13:47:11.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:11 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:11.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:11 smithi077 ceph-mon[22428]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:47:11.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:11 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:11.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:11 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:11.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:11 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:47:11.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:11 smithi077 ceph-mon[22428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:47:11.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:11 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi170.front.sepia.ceph.com:9095"}]: dispatch 2024-05-06T13:47:11.804 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:11 smithi077 ceph-mon[22428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi170.front.sepia.ceph.com:9095"}]: dispatch 2024-05-06T13:47:11.804 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:11 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:11.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:11 smithi170 ceph-mon[23272]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:47:11.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:11 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:11.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:11 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:11.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:11 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:47:11.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:11 smithi170 ceph-mon[23272]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:47:11.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:11 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi170.front.sepia.ceph.com:9095"}]: dispatch 2024-05-06T13:47:11.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:11 smithi170 ceph-mon[23272]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi170.front.sepia.ceph.com:9095"}]: dispatch 2024-05-06T13:47:11.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:11 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:13.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:13 smithi077 ceph-mon[22428]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:47:13.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:13 smithi077 ceph-mon[27166]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:47:13.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:13 smithi170 ceph-mon[23272]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:47:15.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:15 smithi077 ceph-mon[22428]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-05-06T13:47:15.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:15 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:15.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:15 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:15.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:15 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:47:15.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:15 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:47:15.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:15 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:15.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:15 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:47:15.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:15 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:47:15.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:15 smithi077 ceph-mon[27166]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-05-06T13:47:15.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:15 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:15.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:15 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:15.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:15 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:47:15.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:15 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:47:15.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:15 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:15.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:15 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:47:15.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:15 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:47:15.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:15 smithi170 ceph-mon[23272]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-05-06T13:47:15.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:15 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:15.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:15 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:15.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:15 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:47:15.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:15 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:47:15.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:15 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:15.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:15 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:47:15.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:15 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:47:16.777 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:16 smithi077 ceph-mon[22428]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:47:16.777 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:16 smithi077 ceph-mon[22428]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:47:16.777 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:16 smithi077 ceph-mon[27166]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:47:16.778 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:16 smithi077 ceph-mon[27166]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:47:16.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:16 smithi170 ceph-mon[23272]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:47:16.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:16 smithi170 ceph-mon[23272]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:47:17.710 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:17 smithi077 ceph-mon[22428]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-05-06T13:47:17.710 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:17 smithi077 ceph-mon[27166]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-05-06T13:47:17.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:17 smithi170 ceph-mon[23272]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-05-06T13:47:19.516 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:19 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:47:19] "GET /metrics HTTP/1.1" 200 37981 "" "Prometheus/2.33.4" 2024-05-06T13:47:19.775 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:19 smithi077 ceph-mon[22428]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:47:19.776 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:19 smithi077 ceph-mon[27166]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:47:19.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:19 smithi170 ceph-mon[23272]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:47:21.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:21 smithi077 ceph-mon[27166]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:47:21.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:21 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:21.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:21 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:21.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:21 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:47:21.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:21 smithi077 ceph-mon[27166]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:47:21.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:21 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:47:21.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:21 smithi077 ceph-mon[27166]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:47:21.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:21 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3597909852' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:47:21.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:21 smithi077 ceph-mon[22428]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:47:21.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:21 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:21.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:21 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:21.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:21 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:47:21.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:21 smithi077 ceph-mon[22428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:47:21.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:21 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:47:21.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:21 smithi077 ceph-mon[22428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:47:21.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:21 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3597909852' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:47:21.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:21 smithi170 ceph-mon[23272]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:47:21.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:21 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:21.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:21 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:21.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:21 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:47:21.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:21 smithi170 ceph-mon[23272]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:47:21.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:21 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:47:21.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:21 smithi170 ceph-mon[23272]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:47:21.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:21 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/3597909852' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:47:23.517 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:23 smithi077 ceph-mon[22428]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-05-06T13:47:23.518 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:23 smithi077 ceph-mon[27166]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-05-06T13:47:23.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:23 smithi170 ceph-mon[23272]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-05-06T13:47:24.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:24 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:47:24.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:24 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:47:24.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:24 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:47:26.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:25 smithi077 ceph-mon[22428]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-05-06T13:47:26.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:25 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:26.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:25 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:26.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:25 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:47:26.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:25 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:47:26.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:25 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:26.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:25 smithi077 ceph-mon[27166]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-05-06T13:47:26.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:25 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:26.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:25 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:26.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:25 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:47:26.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:25 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:47:26.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:25 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:26.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:25 smithi170 ceph-mon[23272]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-05-06T13:47:26.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:25 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:26.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:25 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:26.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:25 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:47:26.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:25 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:47:26.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:25 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:47:28.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:27 smithi077 ceph-mon[22428]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-05-06T13:47:28.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:27 smithi077 ceph-mon[27166]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-05-06T13:47:28.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:27 smithi170 ceph-mon[23272]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-05-06T13:47:29.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:29 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:47:29] "GET /metrics HTTP/1.1" 200 37972 "" "Prometheus/2.33.4" 2024-05-06T13:47:30.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:29 smithi077 ceph-mon[22428]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-05-06T13:47:30.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:29 smithi077 ceph-mon[27166]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-05-06T13:47:30.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:29 smithi170 ceph-mon[23272]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-05-06T13:47:31.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:30 smithi077 ceph-mon[22428]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-05-06T13:47:31.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:30 smithi077 ceph-mon[22428]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:47:31.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:30 smithi077 ceph-mon[27166]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-05-06T13:47:31.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:30 smithi077 ceph-mon[27166]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:47:31.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:30 smithi170 ceph-mon[23272]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-05-06T13:47:31.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:30 smithi170 ceph-mon[23272]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:47:33.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:32 smithi077 ceph-mon[22428]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-05-06T13:47:33.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:32 smithi077 ceph-mon[27166]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-05-06T13:47:33.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:32 smithi170 ceph-mon[23272]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-05-06T13:47:35.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:34 smithi077 ceph-mon[22428]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:47:35.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:34 smithi077 ceph-mon[27166]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:47:35.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:34 smithi170 ceph-mon[23272]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:47:37.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:36 smithi077 ceph-mon[22428]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:47:37.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:36 smithi077 ceph-mon[27166]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:47:37.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:36 smithi170 ceph-mon[23272]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:47:39.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:38 smithi170 ceph-mon[23272]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:47:39.226 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:38 smithi077 ceph-mon[22428]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:47:39.227 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:38 smithi077 ceph-mon[27166]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:47:39.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:39 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:47:39] "GET /metrics HTTP/1.1" 200 37981 "" "Prometheus/2.33.4" 2024-05-06T13:47:40.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:39 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:47:40.279 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:39 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:47:40.280 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:39 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:47:41.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:40 smithi170 ceph-mon[23272]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:47:41.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:40 smithi170 ceph-mon[23272]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:47:41.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:40 smithi077 ceph-mon[27166]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:47:41.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:40 smithi077 ceph-mon[27166]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:47:41.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:40 smithi077 ceph-mon[22428]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:47:41.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:40 smithi077 ceph-mon[22428]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:47:43.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:42 smithi170 ceph-mon[23272]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:47:43.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:42 smithi077 ceph-mon[22428]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:47:43.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:42 smithi077 ceph-mon[27166]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:47:45.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:44 smithi170 ceph-mon[23272]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:47:45.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:44 smithi077 ceph-mon[22428]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:47:45.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:44 smithi077 ceph-mon[27166]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:47:47.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:46 smithi170 ceph-mon[23272]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:47:47.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:46 smithi077 ceph-mon[22428]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:47:47.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:46 smithi077 ceph-mon[27166]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:47:49.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:48 smithi170 ceph-mon[23272]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:47:49.226 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:48 smithi077 ceph-mon[22428]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:47:49.227 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:48 smithi077 ceph-mon[27166]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:47:49.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:49 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:47:49] "GET /metrics HTTP/1.1" 200 37981 "" "Prometheus/2.33.4" 2024-05-06T13:47:51.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:50 smithi077 ceph-mon[22428]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:47:51.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:50 smithi077 ceph-mon[22428]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:47:51.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:50 smithi077 ceph-mon[27166]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:47:51.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:50 smithi077 ceph-mon[27166]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:47:51.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:50 smithi170 ceph-mon[23272]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:47:51.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:50 smithi170 ceph-mon[23272]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:47:53.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:52 smithi077 ceph-mon[22428]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:47:53.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:52 smithi077 ceph-mon[27166]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:47:53.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:52 smithi170 ceph-mon[23272]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:47:55.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:54 smithi170 ceph-mon[23272]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:47:55.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:54 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:47:55.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:54 smithi077 ceph-mon[22428]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:47:55.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:54 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:47:55.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:54 smithi077 ceph-mon[27166]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:47:55.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:54 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:47:57.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:56 smithi170 ceph-mon[23272]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:47:57.105 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:56 smithi077 ceph-mon[22428]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:47:57.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:56 smithi077 ceph-mon[27166]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:47:59.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:47:58 smithi170 ceph-mon[23272]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:47:59.227 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:47:58 smithi077 ceph-mon[22428]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:47:59.227 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:47:58 smithi077 ceph-mon[27166]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:47:59.553 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:47:59 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:47:59] "GET /metrics HTTP/1.1" 200 37979 "" "Prometheus/2.33.4" 2024-05-06T13:48:01.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:00 smithi077 ceph-mon[22428]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:01.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:00 smithi077 ceph-mon[22428]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:48:01.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:00 smithi077 ceph-mon[27166]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:01.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:00 smithi077 ceph-mon[27166]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:48:01.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:00 smithi170 ceph-mon[23272]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:01.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:00 smithi170 ceph-mon[23272]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:48:03.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:02 smithi170 ceph-mon[23272]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:03.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:02 smithi077 ceph-mon[22428]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:03.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:02 smithi077 ceph-mon[27166]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:05.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:04 smithi170 ceph-mon[23272]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:05.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:04 smithi077 ceph-mon[22428]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:05.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:04 smithi077 ceph-mon[27166]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:07.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:06 smithi170 ceph-mon[23272]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:07.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:06 smithi077 ceph-mon[22428]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:07.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:06 smithi077 ceph-mon[27166]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:09.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:08 smithi077 ceph-mon[22428]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:09.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:08 smithi077 ceph-mon[27166]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:09.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:08 smithi170 ceph-mon[23272]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:09.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:48:09 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:48:09] "GET /metrics HTTP/1.1" 200 37977 "" "Prometheus/2.33.4" 2024-05-06T13:48:10.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:09 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:48:10.299 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:09 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:48:10.299 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:09 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:48:11.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:10 smithi170 ceph-mon[23272]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:11.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:10 smithi170 ceph-mon[23272]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:48:11.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:10 smithi077 ceph-mon[22428]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:11.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:10 smithi077 ceph-mon[22428]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:48:11.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:10 smithi077 ceph-mon[27166]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:11.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:10 smithi077 ceph-mon[27166]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:48:13.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:12 smithi170 ceph-mon[23272]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:13.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:12 smithi077 ceph-mon[22428]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:13.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:12 smithi077 ceph-mon[27166]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:15.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:14 smithi170 ceph-mon[23272]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:15.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:14 smithi077 ceph-mon[22428]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:15.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:14 smithi077 ceph-mon[27166]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:17.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:16 smithi077 ceph-mon[22428]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:17.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:16 smithi077 ceph-mon[27166]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:17.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:16 smithi170 ceph-mon[23272]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:19.226 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:18 smithi077 ceph-mon[22428]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:19.227 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:18 smithi077 ceph-mon[27166]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:19.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:18 smithi170 ceph-mon[23272]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:19.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:48:19 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:48:19] "GET /metrics HTTP/1.1" 200 37977 "" "Prometheus/2.33.4" 2024-05-06T13:48:21.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:20 smithi077 ceph-mon[27166]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:21.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:20 smithi077 ceph-mon[27166]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:48:21.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:20 smithi077 ceph-mon[22428]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:21.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:20 smithi077 ceph-mon[22428]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:48:21.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:20 smithi170 ceph-mon[23272]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:21.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:20 smithi170 ceph-mon[23272]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:48:23.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:22 smithi170 ceph-mon[23272]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:23.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:22 smithi077 ceph-mon[22428]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:23.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:22 smithi077 ceph-mon[27166]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:25.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:24 smithi170 ceph-mon[23272]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:25.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:24 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:48:25.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:24 smithi077 ceph-mon[22428]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:25.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:24 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:48:25.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:24 smithi077 ceph-mon[27166]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:25.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:24 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:48:26.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:26 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:48:26.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:26 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:48:26.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:26 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:48:26.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:26 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:48:26.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:26 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:48:26.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:26 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:48:26.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:26 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:48:26.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:26 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:48:26.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:26 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:48:26.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:26 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:48:26.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:26 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:48:26.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:26 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:48:26.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:26 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:48:26.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:26 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:48:26.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:26 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:48:27.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:27 smithi077 ceph-mon[22428]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:27.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:27 smithi077 ceph-mon[27166]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:27.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:27 smithi170 ceph-mon[23272]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:29.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:29 smithi077 ceph-mon[22428]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:29.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:29 smithi077 ceph-mon[27166]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:29.553 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:48:29 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:48:29] "GET /metrics HTTP/1.1" 200 37977 "" "Prometheus/2.33.4" 2024-05-06T13:48:29.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:29 smithi170 ceph-mon[23272]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:31.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:30 smithi077 ceph-mon[22428]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:31.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:30 smithi077 ceph-mon[22428]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:48:31.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:30 smithi077 ceph-mon[27166]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:31.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:30 smithi077 ceph-mon[27166]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:48:31.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:30 smithi170 ceph-mon[23272]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:31.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:30 smithi170 ceph-mon[23272]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:48:33.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:32 smithi170 ceph-mon[23272]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:33.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:32 smithi077 ceph-mon[22428]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:33.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:32 smithi077 ceph-mon[27166]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:35.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:34 smithi170 ceph-mon[23272]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:35.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:34 smithi077 ceph-mon[22428]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:35.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:34 smithi077 ceph-mon[27166]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:37.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:36 smithi170 ceph-mon[23272]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:37.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:36 smithi077 ceph-mon[22428]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:37.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:36 smithi077 ceph-mon[27166]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:39.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:38 smithi170 ceph-mon[23272]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:39.227 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:38 smithi077 ceph-mon[22428]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:39.227 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:38 smithi077 ceph-mon[27166]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:39.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:48:39 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:48:39] "GET /metrics HTTP/1.1" 200 37973 "" "Prometheus/2.33.4" 2024-05-06T13:48:40.225 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:39 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:48:40.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:39 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:48:40.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:39 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:48:41.125 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:40 smithi170 ceph-mon[23272]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:41.126 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:40 smithi170 ceph-mon[23272]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:48:41.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:40 smithi077 ceph-mon[22428]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:41.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:40 smithi077 ceph-mon[22428]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:48:41.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:40 smithi077 ceph-mon[27166]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:41.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:40 smithi077 ceph-mon[27166]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:48:43.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:42 smithi077 ceph-mon[22428]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:43.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:42 smithi077 ceph-mon[27166]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:43.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:42 smithi170 ceph-mon[23272]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:45.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:44 smithi077 ceph-mon[22428]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:45.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:44 smithi077 ceph-mon[27166]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:45.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:44 smithi170 ceph-mon[23272]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:47.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:46 smithi077 ceph-mon[22428]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:47.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:46 smithi077 ceph-mon[27166]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:47.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:46 smithi170 ceph-mon[23272]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:49.226 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:48 smithi077 ceph-mon[22428]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:49.226 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:48 smithi077 ceph-mon[27166]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:49.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:48 smithi170 ceph-mon[23272]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:49.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:48:49 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:48:49] "GET /metrics HTTP/1.1" 200 37973 "" "Prometheus/2.33.4" 2024-05-06T13:48:51.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:50 smithi077 ceph-mon[22428]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:51.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:50 smithi077 ceph-mon[22428]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:48:51.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:50 smithi077 ceph-mon[27166]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:51.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:50 smithi077 ceph-mon[27166]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:48:51.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:50 smithi170 ceph-mon[23272]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:51.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:50 smithi170 ceph-mon[23272]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:48:53.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:52 smithi170 ceph-mon[23272]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:53.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:52 smithi077 ceph-mon[22428]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:53.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:52 smithi077 ceph-mon[27166]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:55.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:54 smithi170 ceph-mon[23272]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:55.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:54 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:48:55.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:54 smithi077 ceph-mon[22428]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:55.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:54 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:48:55.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:54 smithi077 ceph-mon[27166]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:55.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:54 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:48:57.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:56 smithi170 ceph-mon[23272]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:57.105 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:56 smithi077 ceph-mon[22428]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:57.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:56 smithi077 ceph-mon[27166]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:48:59.227 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:48:58 smithi077 ceph-mon[22428]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:59.227 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:48:58 smithi077 ceph-mon[27166]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:59.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:48:58 smithi170 ceph-mon[23272]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:48:59.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:48:59 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:48:59] "GET /metrics HTTP/1.1" 200 37976 "" "Prometheus/2.33.4" 2024-05-06T13:49:01.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:00 smithi077 ceph-mon[22428]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:01.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:00 smithi077 ceph-mon[22428]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:49:01.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:00 smithi077 ceph-mon[27166]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:01.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:00 smithi077 ceph-mon[27166]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:49:01.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:00 smithi170 ceph-mon[23272]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:01.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:00 smithi170 ceph-mon[23272]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:49:03.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:02 smithi170 ceph-mon[23272]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:03.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:02 smithi077 ceph-mon[22428]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:03.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:02 smithi077 ceph-mon[27166]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:05.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:04 smithi077 ceph-mon[22428]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:05.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:04 smithi077 ceph-mon[27166]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:05.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:04 smithi170 ceph-mon[23272]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:07.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:06 smithi077 ceph-mon[22428]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:07.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:06 smithi077 ceph-mon[27166]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:07.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:06 smithi170 ceph-mon[23272]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:09.128 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:08 smithi170 ceph-mon[23272]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:09.128 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:08 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:49:09.227 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:08 smithi077 ceph-mon[22428]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:09.227 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:08 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:49:09.227 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:08 smithi077 ceph-mon[27166]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:09.227 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:08 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:49:09.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:49:09 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:49:09] "GET /metrics HTTP/1.1" 200 37987 "" "Prometheus/2.33.4" 2024-05-06T13:49:11.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:10 smithi077 ceph-mon[22428]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:11.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:10 smithi077 ceph-mon[22428]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:49:11.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:10 smithi077 ceph-mon[27166]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:11.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:10 smithi077 ceph-mon[27166]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:49:11.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:10 smithi170 ceph-mon[23272]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:11.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:10 smithi170 ceph-mon[23272]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:49:13.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:12 smithi077 ceph-mon[22428]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:13.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:12 smithi077 ceph-mon[27166]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:13.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:12 smithi170 ceph-mon[23272]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:15.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:14 smithi077 ceph-mon[27166]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:15.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:14 smithi077 ceph-mon[22428]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:15.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:14 smithi170 ceph-mon[23272]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:17.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:16 smithi077 ceph-mon[22428]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:17.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:16 smithi077 ceph-mon[27166]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:17.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:16 smithi170 ceph-mon[23272]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:19.227 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:18 smithi077 ceph-mon[22428]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:19.227 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:18 smithi077 ceph-mon[27166]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:19.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:18 smithi170 ceph-mon[23272]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:19.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:49:19 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:49:19] "GET /metrics HTTP/1.1" 200 37987 "" "Prometheus/2.33.4" 2024-05-06T13:49:21.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:20 smithi077 ceph-mon[22428]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:21.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:20 smithi077 ceph-mon[22428]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:49:21.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:20 smithi077 ceph-mon[27166]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:21.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:20 smithi077 ceph-mon[27166]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:49:21.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:20 smithi170 ceph-mon[23272]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:21.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:20 smithi170 ceph-mon[23272]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:49:23.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:22 smithi077 ceph-mon[22428]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:23.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:22 smithi077 ceph-mon[27166]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:23.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:22 smithi170 ceph-mon[23272]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:24.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:23 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:49:24.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:23 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:49:24.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:23 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:49:25.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:24 smithi077 ceph-mon[22428]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:25.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:24 smithi077 ceph-mon[27166]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:25.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:24 smithi170 ceph-mon[23272]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:27.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:27 smithi077 ceph-mon[22428]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:27.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:27 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:49:27.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:27 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:49:27.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:27 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:49:27.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:27 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:49:27.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:27 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:49:27.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:27 smithi077 ceph-mon[27166]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:27.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:27 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:49:27.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:27 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:49:27.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:27 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:49:27.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:27 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:49:27.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:27 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:49:27.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:27 smithi170 ceph-mon[23272]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:27.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:27 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:49:27.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:27 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:49:27.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:27 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:49:27.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:27 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:49:27.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:27 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' 2024-05-06T13:49:29.486 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:49:29 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:49:29] "GET /metrics HTTP/1.1" 200 37986 "" "Prometheus/2.33.4" 2024-05-06T13:49:29.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:29 smithi077 ceph-mon[22428]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:29.805 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:29 smithi077 ceph-mon[27166]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:29.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:29 smithi170 ceph-mon[23272]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:31.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:30 smithi077 ceph-mon[27166]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:31.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:30 smithi077 ceph-mon[27166]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:49:31.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:30 smithi077 ceph-mon[22428]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:31.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:30 smithi077 ceph-mon[22428]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:49:31.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:30 smithi170 ceph-mon[23272]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:31.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:30 smithi170 ceph-mon[23272]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:49:33.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:32 smithi077 ceph-mon[22428]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:33.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:32 smithi077 ceph-mon[27166]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:33.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:32 smithi170 ceph-mon[23272]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:35.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:34 smithi077 ceph-mon[22428]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:35.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:34 smithi077 ceph-mon[27166]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:35.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:34 smithi170 ceph-mon[23272]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:37.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:36 smithi077 ceph-mon[27166]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:37.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:36 smithi077 ceph-mon[22428]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:37.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:36 smithi170 ceph-mon[23272]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:39.129 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:38 smithi170 ceph-mon[23272]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:39.129 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:38 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:49:39.227 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:38 smithi077 ceph-mon[22428]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:39.227 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:38 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:49:39.227 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:38 smithi077 ceph-mon[27166]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:39.227 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:38 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:49:39.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:49:39 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:49:39] "GET /metrics HTTP/1.1" 200 37986 "" "Prometheus/2.33.4" 2024-05-06T13:49:41.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:40 smithi077 ceph-mon[22428]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:41.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:40 smithi077 ceph-mon[22428]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:49:41.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:40 smithi077 ceph-mon[27166]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:41.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:40 smithi077 ceph-mon[27166]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:49:41.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:40 smithi170 ceph-mon[23272]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:41.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:40 smithi170 ceph-mon[23272]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:49:43.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:42 smithi077 ceph-mon[22428]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:43.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:42 smithi077 ceph-mon[27166]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:43.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:42 smithi170 ceph-mon[23272]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:45.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:44 smithi077 ceph-mon[27166]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:45.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:44 smithi077 ceph-mon[22428]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:45.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:44 smithi170 ceph-mon[23272]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:47.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:46 smithi077 ceph-mon[22428]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:47.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:46 smithi077 ceph-mon[27166]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:47.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:46 smithi170 ceph-mon[23272]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:49.227 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:48 smithi077 ceph-mon[22428]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:49.227 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:48 smithi077 ceph-mon[27166]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:49.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:48 smithi170 ceph-mon[23272]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:49.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:49:49 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:49:49] "GET /metrics HTTP/1.1" 200 37986 "" "Prometheus/2.33.4" 2024-05-06T13:49:51.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:50 smithi077 ceph-mon[27166]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:51.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:50 smithi077 ceph-mon[27166]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:49:51.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:50 smithi077 ceph-mon[22428]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:51.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:50 smithi077 ceph-mon[22428]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:49:51.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:50 smithi170 ceph-mon[23272]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:51.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:50 smithi170 ceph-mon[23272]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:49:53.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:52 smithi077 ceph-mon[22428]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:53.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:52 smithi077 ceph-mon[27166]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:53.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:52 smithi170 ceph-mon[23272]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:54.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:53 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:49:54.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:53 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:49:54.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:53 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:49:55.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:54 smithi077 ceph-mon[22428]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:55.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:54 smithi077 ceph-mon[27166]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:55.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:54 smithi170 ceph-mon[23272]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:57.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:56 smithi077 ceph-mon[22428]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:57.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:56 smithi077 ceph-mon[27166]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:57.136 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch ps' 2024-05-06T13:49:57.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:56 smithi170 ceph-mon[23272]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:49:59.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:58 smithi077 ceph-mon[22428]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:59.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:58 smithi077 ceph-mon[27166]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:59.181 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T13:49:59.182 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (9m) 2m ago 17m 23.4M - 0.23.0 ba2b418f427c 0f855548777f 2024-05-06T13:49:59.182 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (8m) 2m ago 16m 53.8M - 8.3.5 dad864ee21e9 fdcecac79925 2024-05-06T13:49:59.182 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (2m) 2m ago 16m 45.9M - 3.5 e1d6a67b021e ded1b53a36b8 2024-05-06T13:49:59.182 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283 running (12m) 2m ago 21m 448M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 63ef6825ff68 2024-05-06T13:49:59.182 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (5m) 2m ago 23m 501M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T13:49:59.182 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (23m) 2m ago 23m 62.7M 2048M 17.2.0 e1d6a67b021e ac46f91694f8 2024-05-06T13:49:59.182 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (21m) 2m ago 21m 47.0M 2048M 17.2.0 e1d6a67b021e 24be6fce9d28 2024-05-06T13:49:59.182 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (22m) 2m ago 22m 46.0M 2048M 17.2.0 e1d6a67b021e 83af336ef762 2024-05-06T13:49:59.182 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (9m) 2m ago 17m 17.4M - 1.3.1 1dbe0e931976 54aee5b22b1c 2024-05-06T13:49:59.182 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (9m) 2m ago 17m 18.4M - 1.3.1 1dbe0e931976 ca5df43a7469 2024-05-06T13:49:59.182 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (21m) 2m ago 21m 52.9M 2992M 17.2.0 e1d6a67b021e 2ccee2bb6e96 2024-05-06T13:49:59.182 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (20m) 2m ago 20m 56.9M 2992M 17.2.0 e1d6a67b021e 1d9e6609b585 2024-05-06T13:49:59.182 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (20m) 2m ago 20m 51.8M 2992M 17.2.0 e1d6a67b021e 37a61d295111 2024-05-06T13:49:59.183 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (20m) 2m ago 20m 51.2M 2992M 17.2.0 e1d6a67b021e 8feb7c077aaa 2024-05-06T13:49:59.183 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (19m) 2m ago 19m 54.9M 3248M 17.2.0 e1d6a67b021e f865acc44a06 2024-05-06T13:49:59.183 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (19m) 2m ago 19m 51.3M 3248M 17.2.0 e1d6a67b021e a55bd4122f6f 2024-05-06T13:49:59.183 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (19m) 2m ago 19m 50.1M 3248M 17.2.0 e1d6a67b021e 362938a0b357 2024-05-06T13:49:59.183 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (18m) 2m ago 18m 52.2M 3248M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T13:49:59.183 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (2m) 2m ago 17m 46.7M - 2.33.4 514e6a882f6e 7550480b5a53 2024-05-06T13:49:59.183 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (16m) 2m ago 16m 75.5M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T13:49:59.183 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (16m) 2m ago 16m 74.0M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T13:49:59.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:58 smithi170 ceph-mon[23272]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:49:59.461 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:49:59 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:49:59] "GET /metrics HTTP/1.1" 200 37980 "" "Prometheus/2.33.4" 2024-05-06T13:49:59.870 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph versions' 2024-05-06T13:50:00.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:49:59 smithi077 ceph-mon[22428]: from='client.25285 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:50:00.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:49:59 smithi077 ceph-mon[27166]: from='client.25285 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:50:00.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:49:59 smithi170 ceph-mon[23272]: from='client.25285 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:50:01.110 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:00 smithi077 ceph-mon[22428]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:50:01.110 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:00 smithi077 ceph-mon[22428]: overall HEALTH_OK 2024-05-06T13:50:01.110 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:00 smithi077 ceph-mon[22428]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:50:01.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:00 smithi077 ceph-mon[27166]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:50:01.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:00 smithi077 ceph-mon[27166]: overall HEALTH_OK 2024-05-06T13:50:01.110 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:00 smithi077 ceph-mon[27166]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:50:01.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:00 smithi170 ceph-mon[23272]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:50:01.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:00 smithi170 ceph-mon[23272]: overall HEALTH_OK 2024-05-06T13:50:01.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:00 smithi170 ceph-mon[23272]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:50:01.852 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:50:01.853 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-05-06T13:50:01.853 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-05-06T13:50:01.853 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:50:01.853 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-05-06T13:50:01.853 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T13:50:01.853 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:50:01.853 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-05-06T13:50:01.853 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-06T13:50:01.853 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:50:01.853 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": {}, 2024-05-06T13:50:01.853 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw": { 2024-05-06T13:50:01.853 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T13:50:01.853 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:50:01.853 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-05-06T13:50:01.853 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-05-06T13:50:01.854 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T13:50:01.854 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T13:50:01.854 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:50:02.184 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:01 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3978964551' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:50:02.184 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:01 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3978964551' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:50:02.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:01 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/3978964551' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:50:02.486 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph -s' 2024-05-06T13:50:03.046 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:02 smithi077 ceph-mon[22428]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:50:03.046 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:02 smithi077 ceph-mon[27166]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:50:03.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:02 smithi170 ceph-mon[23272]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:50:04.454 INFO:teuthology.orchestra.run.smithi077.stdout: cluster: 2024-05-06T13:50:04.455 INFO:teuthology.orchestra.run.smithi077.stdout: id: 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:50:04.455 INFO:teuthology.orchestra.run.smithi077.stdout: health: HEALTH_OK 2024-05-06T13:50:04.455 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:50:04.455 INFO:teuthology.orchestra.run.smithi077.stdout: services: 2024-05-06T13:50:04.455 INFO:teuthology.orchestra.run.smithi077.stdout: mon: 3 daemons, quorum a,c,b (age 21m) 2024-05-06T13:50:04.455 INFO:teuthology.orchestra.run.smithi077.stdout: mgr: y(active, since 3m), standbys: x 2024-05-06T13:50:04.455 INFO:teuthology.orchestra.run.smithi077.stdout: osd: 8 osds: 8 up (since 18m), 8 in (since 18m) 2024-05-06T13:50:04.455 INFO:teuthology.orchestra.run.smithi077.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-05-06T13:50:04.455 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:50:04.455 INFO:teuthology.orchestra.run.smithi077.stdout: data: 2024-05-06T13:50:04.455 INFO:teuthology.orchestra.run.smithi077.stdout: pools: 6 pools, 161 pgs 2024-05-06T13:50:04.456 INFO:teuthology.orchestra.run.smithi077.stdout: objects: 209 objects, 585 KiB 2024-05-06T13:50:04.456 INFO:teuthology.orchestra.run.smithi077.stdout: usage: 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:50:04.456 INFO:teuthology.orchestra.run.smithi077.stdout: pgs: 161 active+clean 2024-05-06T13:50:04.456 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:50:04.456 INFO:teuthology.orchestra.run.smithi077.stdout: io: 2024-05-06T13:50:04.456 INFO:teuthology.orchestra.run.smithi077.stdout: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-05-06T13:50:04.456 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-05-06T13:50:05.085 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:04 smithi077 ceph-mon[22428]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:50:05.085 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:04 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3115182035' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-06T13:50:05.085 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:04 smithi077 ceph-mon[27166]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:50:05.086 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:04 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3115182035' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-06T13:50:05.106 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph health detail' 2024-05-06T13:50:05.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:04 smithi170 ceph-mon[23272]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:50:05.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:04 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/3115182035' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-06T13:50:07.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:06 smithi077 ceph-mon[22428]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:50:07.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:06 smithi077 ceph-mon[27166]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:50:07.176 INFO:teuthology.orchestra.run.smithi077.stdout:HEALTH_OK 2024-05-06T13:50:07.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:06 smithi170 ceph-mon[23272]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:50:07.786 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-05-06T13:50:08.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:07 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3029610829' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T13:50:08.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:07 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3029610829' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T13:50:08.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:07 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/3029610829' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T13:50:09.201 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:08 smithi077 ceph-mon[22428]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:50:09.201 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:08 smithi077 ceph-mon[22428]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:50:09.201 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:08 smithi077 ceph-mon[27166]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:50:09.201 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:08 smithi077 ceph-mon[27166]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:50:09.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:08 smithi170 ceph-mon[23272]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:50:09.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:08 smithi170 ceph-mon[23272]: from='mgr.25147 172.21.15.77:0/3254707235' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:50:09.457 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:09 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:50:09] "GET /metrics HTTP/1.1" 200 37981 "" "Prometheus/2.33.4" 2024-05-06T13:50:09.869 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T13:50:10.526 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph mgr fail' 2024-05-06T13:50:10.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:10 smithi077 ceph-mon[22428]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:50:10.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:10 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/4000643247' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:50:10.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:10 smithi077 ceph-mon[22428]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:50:10.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:10 smithi077 ceph-mon[27166]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:50:10.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:10 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/4000643247' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:50:10.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:10 smithi077 ceph-mon[27166]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:50:11.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:10 smithi170 ceph-mon[23272]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:50:11.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:10 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/4000643247' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:50:11.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:10 smithi170 ceph-mon[23272]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:50:13.225 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:12 smithi077 ceph-mon[22428]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:50:13.225 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:12 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/4052651565' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-05-06T13:50:13.225 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:12 smithi077 ceph-mon[22428]: osdmap e105: 8 total, 8 up, 8 in 2024-05-06T13:50:13.226 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:12 smithi077 ceph-mon[27166]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:50:13.226 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:12 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/4052651565' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-05-06T13:50:13.226 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:12 smithi077 ceph-mon[27166]: osdmap e105: 8 total, 8 up, 8 in 2024-05-06T13:50:13.226 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ignoring --setuser ceph since I am not root 2024-05-06T13:50:13.226 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ignoring --setgroup ceph since I am not root 2024-05-06T13:50:13.226 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:12 smithi077 ceph-mgr[60780]: -- 172.21.15.77:0/2364312174 <== mon.0 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55847acfc5a0 con 0x55847acd1000 2024-05-06T13:50:13.226 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:50:13.032+0000 7fb109c02200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T13:50:13.226 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:50:13.224+0000 7fb109c02200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:50:13.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:12 smithi170 ceph-mon[23272]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:50:13.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:12 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/4052651565' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-05-06T13:50:13.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:12 smithi170 ceph-mon[23272]: osdmap e105: 8 total, 8 up, 8 in 2024-05-06T13:50:13.327 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:12 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:50:12] ENGINE Bus STOPPING 2024-05-06T13:50:13.327 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:12 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:50:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-06T13:50:13.327 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:12 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:50:12] ENGINE Bus STOPPED 2024-05-06T13:50:13.327 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:12 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:50:12] ENGINE Bus STARTING 2024-05-06T13:50:13.327 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:13 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:50:13] ENGINE Serving on http://:::9283 2024-05-06T13:50:13.327 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:13 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:50:13] ENGINE Bus STARTED 2024-05-06T13:50:13.536 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:50:13.336+0000 7fb109c02200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T13:50:13.563 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'sleep 180' 2024-05-06T13:50:13.802 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:50:13.668+0000 7fb109c02200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T13:50:13.803 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:50:13.769+0000 7fb109c02200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T13:50:14.173 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/4052651565' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-05-06T13:50:14.173 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[22428]: mgrmap e33: x(active, starting, since 0.214938s) 2024-05-06T13:50:14.173 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:50:14.173 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:50:14.173 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:50:14.173 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T13:50:14.173 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:50:14.174 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:50:14.174 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:50:14.174 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:50:14.174 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:50:14.174 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:50:14.174 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:50:14.174 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:50:14.174 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T13:50:14.174 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T13:50:14.174 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T13:50:14.174 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[22428]: Manager daemon x is now available 2024-05-06T13:50:14.174 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:50:14.174 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:50:14.175 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-06T13:50:14.175 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[22428]: [06/May/2024:13:50:13] ENGINE Bus STARTING 2024-05-06T13:50:14.175 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[22428]: [06/May/2024:13:50:13] ENGINE Serving on https://172.21.15.170:7150 2024-05-06T13:50:14.175 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[22428]: [06/May/2024:13:50:13] ENGINE Serving on http://172.21.15.170:8765 2024-05-06T13:50:14.175 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[22428]: [06/May/2024:13:50:13] ENGINE Bus STARTED 2024-05-06T13:50:14.175 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/4052651565' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-05-06T13:50:14.175 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[27166]: mgrmap e33: x(active, starting, since 0.214938s) 2024-05-06T13:50:14.175 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:50:14.175 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:50:14.176 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:50:14.176 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T13:50:14.176 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:50:14.176 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:50:14.176 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:50:14.176 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:50:14.176 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:50:14.176 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:50:14.176 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:50:14.176 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:50:14.176 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T13:50:14.176 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T13:50:14.176 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T13:50:14.177 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[27166]: Manager daemon x is now available 2024-05-06T13:50:14.177 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:50:14.177 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:50:14.177 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-06T13:50:14.177 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[27166]: [06/May/2024:13:50:13] ENGINE Bus STARTING 2024-05-06T13:50:14.177 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[27166]: [06/May/2024:13:50:13] ENGINE Serving on https://172.21.15.170:7150 2024-05-06T13:50:14.177 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[27166]: [06/May/2024:13:50:13] ENGINE Serving on http://172.21.15.170:8765 2024-05-06T13:50:14.177 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:13 smithi077 ceph-mon[27166]: [06/May/2024:13:50:13] ENGINE Bus STARTED 2024-05-06T13:50:14.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:13 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/4052651565' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-05-06T13:50:14.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:13 smithi170 ceph-mon[23272]: mgrmap e33: x(active, starting, since 0.214938s) 2024-05-06T13:50:14.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:13 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:50:14.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:13 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:50:14.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:13 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:50:14.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:13 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T13:50:14.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:13 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:50:14.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:13 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:50:14.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:13 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:50:14.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:13 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:50:14.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:13 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:50:14.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:13 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:50:14.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:13 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:50:14.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:13 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:50:14.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:13 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T13:50:14.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:13 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T13:50:14.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:13 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T13:50:14.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:13 smithi170 ceph-mon[23272]: Manager daemon x is now available 2024-05-06T13:50:14.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:13 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:50:14.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:13 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:50:14.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:13 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-06T13:50:14.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:13 smithi170 ceph-mon[23272]: [06/May/2024:13:50:13] ENGINE Bus STARTING 2024-05-06T13:50:14.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:13 smithi170 ceph-mon[23272]: [06/May/2024:13:50:13] ENGINE Serving on https://172.21.15.170:7150 2024-05-06T13:50:14.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:13 smithi170 ceph-mon[23272]: [06/May/2024:13:50:13] ENGINE Serving on http://172.21.15.170:8765 2024-05-06T13:50:14.329 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:13 smithi170 ceph-mon[23272]: [06/May/2024:13:50:13] ENGINE Bus STARTED 2024-05-06T13:50:14.931 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:14 smithi077 ceph-mon[22428]: mgrmap e34: x(active, since 1.22204s) 2024-05-06T13:50:14.931 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:14 smithi077 ceph-mon[22428]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:50:14.931 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:14 smithi077 ceph-mon[27166]: mgrmap e34: x(active, since 1.22204s) 2024-05-06T13:50:14.932 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:14 smithi077 ceph-mon[27166]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:50:15.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:14 smithi170 ceph-mon[23272]: mgrmap e34: x(active, since 1.22204s) 2024-05-06T13:50:15.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:14 smithi170 ceph-mon[23272]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:50:15.265 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:14 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:50:14.992+0000 7fb109c02200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T13:50:15.266 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:15 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:50:15.170+0000 7fb109c02200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T13:50:15.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:15 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:50:15.346+0000 7fb109c02200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T13:50:15.960 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:15 smithi077 ceph-mon[22428]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:50:15.961 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:15 smithi077 ceph-mon[27166]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:50:15.961 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:15 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:50:15.644+0000 7fb109c02200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T13:50:15.961 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:15 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:50:15.751+0000 7fb109c02200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T13:50:16.197 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:15 smithi170 ceph-mon[23272]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:50:16.302 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:15 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:50:15.959+0000 7fb109c02200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T13:50:16.302 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:16 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:50:16.159+0000 7fb109c02200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:50:17.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:16 smithi077 ceph-mon[22428]: mgrmap e35: x(active, since 3s) 2024-05-06T13:50:17.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:16 smithi077 ceph-mon[27166]: mgrmap e35: x(active, since 3s) 2024-05-06T13:50:17.053 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:16 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:50:16.794+0000 7fb109c02200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T13:50:17.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:16 smithi170 ceph-mon[23272]: mgrmap e35: x(active, since 3s) 2024-05-06T13:50:17.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:17 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:50:17.252+0000 7fb109c02200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T13:50:17.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:17 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:50:17.354+0000 7fb109c02200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T13:50:17.901 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:17 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:50:17.562+0000 7fb109c02200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-06T13:50:18.205 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:18 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:50:18.107+0000 7fb109c02200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T13:50:18.469 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:18 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:50:18.204+0000 7fb109c02200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T13:50:18.469 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:18 smithi077 ceph-mon[22428]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:50:18.469 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:18 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:18.469 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:18 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:18.469 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:18 smithi077 ceph-mon[27166]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:50:18.470 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:18 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:18.470 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:18 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:18.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:18 smithi170 ceph-mon[23272]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:50:18.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:18 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:18.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:18 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:18.781 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:18 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:50:18.468+0000 7fb109c02200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T13:50:18.782 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:18 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:50:18.605+0000 7fb109c02200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T13:50:18.782 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:18 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:50:18.706+0000 7fb109c02200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T13:50:19.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:18 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:50:18.830+0000 7fb109c02200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T13:50:19.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:18 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:50:18.925+0000 7fb109c02200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T13:50:19.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:19 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:19.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:19 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:19.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:19 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:19.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:19 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:19.784 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:19 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:19.784 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:19 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:19.785 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:19 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:19.785 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:19 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:19.785 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:19 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:19.785 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:19 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:19.785 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:19 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:19.785 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:19 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:20.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:19 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:50:19.783+0000 7fb109c02200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T13:50:20.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:20 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:50:20.017+0000 7fb109c02200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T13:50:20.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:20 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:50:20] ENGINE Bus STARTING 2024-05-06T13:50:20.406 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:20 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: CherryPy Checker: 2024-05-06T13:50:20.556 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:20 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: The Application mounted at '' has an empty config. 2024-05-06T13:50:20.557 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:20 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:50:20.557 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:20 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:50:20] ENGINE Serving on http://:::9283 2024-05-06T13:50:20.557 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:50:20 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:50:20] ENGINE Bus STARTED 2024-05-06T13:50:21.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:20 smithi077 ceph-mon[22428]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:50:21.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:20 smithi077 ceph-mon[22428]: Standby manager daemon y started 2024-05-06T13:50:21.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:20 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-06T13:50:21.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:20 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:50:21.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:20 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-06T13:50:21.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:20 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:50:21.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:20 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:21.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:20 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:21.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:20 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:50:21.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:20 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:50:21.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:20 smithi077 ceph-mon[27166]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:50:21.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:20 smithi077 ceph-mon[27166]: Standby manager daemon y started 2024-05-06T13:50:21.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:20 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-06T13:50:21.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:20 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:50:21.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:20 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-06T13:50:21.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:20 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:50:21.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:20 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:21.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:20 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:21.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:20 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:50:21.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:20 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:50:21.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:20 smithi170 ceph-mon[23272]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:50:21.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:20 smithi170 ceph-mon[23272]: Standby manager daemon y started 2024-05-06T13:50:21.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:20 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-06T13:50:21.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:20 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:50:21.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:20 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-06T13:50:21.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:20 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:50:21.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:20 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:21.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:20 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:21.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:20 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:50:21.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:20 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:50:22.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[22428]: Updating smithi077:/etc/ceph/ceph.conf 2024-05-06T13:50:22.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[22428]: Updating smithi170:/etc/ceph/ceph.conf 2024-05-06T13:50:22.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[22428]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:50:22.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[22428]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:50:22.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[22428]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:50:22.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[22428]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:50:22.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[22428]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:50:22.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[22428]: mgrmap e36: x(active, since 8s), standbys: y 2024-05-06T13:50:22.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T13:50:22.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:22.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:22.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:22.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:22.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:22.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:50:22.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:50:22.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[27166]: Updating smithi077:/etc/ceph/ceph.conf 2024-05-06T13:50:22.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[27166]: Updating smithi170:/etc/ceph/ceph.conf 2024-05-06T13:50:22.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[27166]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:50:22.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[27166]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:50:22.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[27166]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:50:22.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[27166]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:50:22.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[27166]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:50:22.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[27166]: mgrmap e36: x(active, since 8s), standbys: y 2024-05-06T13:50:22.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T13:50:22.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:22.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:22.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:22.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:22.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:22.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:50:22.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:21 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:50:22.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:21 smithi170 ceph-mon[23272]: Updating smithi077:/etc/ceph/ceph.conf 2024-05-06T13:50:22.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:21 smithi170 ceph-mon[23272]: Updating smithi170:/etc/ceph/ceph.conf 2024-05-06T13:50:22.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:21 smithi170 ceph-mon[23272]: from='client.15369 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:50:22.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:21 smithi170 ceph-mon[23272]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:50:22.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:21 smithi170 ceph-mon[23272]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:50:22.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:21 smithi170 ceph-mon[23272]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:50:22.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:21 smithi170 ceph-mon[23272]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:50:22.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:21 smithi170 ceph-mon[23272]: mgrmap e36: x(active, since 8s), standbys: y 2024-05-06T13:50:22.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:21 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T13:50:22.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:21 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:22.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:21 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:22.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:21 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:22.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:21 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:22.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:21 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:22.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:21 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:50:22.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:21 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:50:22.993 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:22 smithi077 ceph-mon[22428]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-05-06T13:50:22.993 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:22 smithi077 ceph-mon[22428]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:50:22.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:22 smithi077 ceph-mon[22428]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:50:22.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:22 smithi077 ceph-mon[22428]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:50:22.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:22 smithi077 ceph-mon[22428]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:50:22.994 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:22 smithi077 ceph-mon[27166]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-05-06T13:50:22.994 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:22 smithi077 ceph-mon[27166]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:50:22.994 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:22 smithi077 ceph-mon[27166]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:50:22.994 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:22 smithi077 ceph-mon[27166]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:50:22.994 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:22 smithi077 ceph-mon[27166]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:50:23.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:22 smithi170 ceph-mon[23272]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-05-06T13:50:23.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:22 smithi170 ceph-mon[23272]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:50:23.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:22 smithi170 ceph-mon[23272]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:50:23.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:22 smithi170 ceph-mon[23272]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:50:23.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:22 smithi170 ceph-mon[23272]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:50:24.737 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:50:24.449+0000 7f5694f4b700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:50:24.737 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Creating ceph-iscsi config... 2024-05-06T13:50:24.737 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:50:24.737 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:50:24.737 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:50:24.737 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:50:24.737 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:50:24.737 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Traceback (most recent call last): 2024-05-06T13:50:24.737 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:50:24.737 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return _run_code(code, main_globals, None, 2024-05-06T13:50:24.738 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:50:24.738 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: exec(code, run_globals) 2024-05-06T13:50:24.738 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:50:24.738 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:50:24.738 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:50:24.738 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:50:24.738 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:50:24.738 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:50:24.738 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:50:24.739 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:50:24.739 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:50:24.739 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Traceback (most recent call last): 2024-05-06T13:50:24.739 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:50:24.739 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:50:24.739 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:50:24.739 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:50:24.739 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:50:24.739 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:50:24.739 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:50:24.739 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return future.result(timeout) 2024-05-06T13:50:24.739 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:50:24.739 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return self.__get_result() 2024-05-06T13:50:24.739 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:50:24.740 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: raise self._exception 2024-05-06T13:50:24.740 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:50:24.740 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: use_current_daemon_image=reconfig, 2024-05-06T13:50:24.740 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:50:24.740 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:50:24.740 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:50:24.740 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Creating ceph-iscsi config... 2024-05-06T13:50:24.740 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:50:24.740 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:50:24.740 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:50:24.740 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:50:24.740 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:50:24.740 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: Traceback (most recent call last): 2024-05-06T13:50:24.740 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:50:24.740 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: return _run_code(code, main_globals, None, 2024-05-06T13:50:24.741 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:50:24.741 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: exec(code, run_globals) 2024-05-06T13:50:24.741 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:50:24.741 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:50:24.741 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:50:24.741 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:50:24.741 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:50:24.741 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:50:24.741 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:50:24.741 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:50:24.741 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:50:25.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:24 smithi077 ceph-mon[22428]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-06T13:50:25.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:24 smithi077 ceph-mon[27166]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-06T13:50:25.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:24 smithi170 ceph-mon[23272]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-06T13:50:25.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:25 smithi170 ceph-mon[23272]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:50:25.827 INFO:journalctl@ceph.mon.b.smithi170.stdout: Creating ceph-iscsi config... 2024-05-06T13:50:25.827 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:50:25.827 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:50:25.827 INFO:journalctl@ceph.mon.b.smithi170.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:50:25.827 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:50:25.827 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:50:25.828 INFO:journalctl@ceph.mon.b.smithi170.stdout: Traceback (most recent call last): 2024-05-06T13:50:25.828 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:50:25.828 INFO:journalctl@ceph.mon.b.smithi170.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:50:25.828 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:50:25.828 INFO:journalctl@ceph.mon.b.smithi170.stdout: exec(code, run_globals) 2024-05-06T13:50:25.828 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:50:25.829 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:50:25.829 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:50:25.829 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:50:25.829 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:50:25.829 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:50:25.829 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:50:25.829 INFO:journalctl@ceph.mon.b.smithi170.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:50:25.829 INFO:journalctl@ceph.mon.b.smithi170.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:50:25.829 INFO:journalctl@ceph.mon.b.smithi170.stdout: Traceback (most recent call last): 2024-05-06T13:50:25.829 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:50:25.829 INFO:journalctl@ceph.mon.b.smithi170.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:50:25.829 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:50:25.830 INFO:journalctl@ceph.mon.b.smithi170.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:50:25.830 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:50:25.830 INFO:journalctl@ceph.mon.b.smithi170.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:50:25.830 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:50:25.830 INFO:journalctl@ceph.mon.b.smithi170.stdout: return future.result(timeout) 2024-05-06T13:50:25.830 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:50:25.830 INFO:journalctl@ceph.mon.b.smithi170.stdout: return self.__get_result() 2024-05-06T13:50:25.830 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:50:25.830 INFO:journalctl@ceph.mon.b.smithi170.stdout: raise self._exception 2024-05-06T13:50:25.830 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:50:25.830 INFO:journalctl@ceph.mon.b.smithi170.stdout: use_current_daemon_image=reconfig, 2024-05-06T13:50:25.831 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:50:25.831 INFO:journalctl@ceph.mon.b.smithi170.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:50:25.831 INFO:journalctl@ceph.mon.b.smithi170.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:50:25.831 INFO:journalctl@ceph.mon.b.smithi170.stdout: Creating ceph-iscsi config... 2024-05-06T13:50:25.831 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:50:25.831 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:50:25.831 INFO:journalctl@ceph.mon.b.smithi170.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:50:25.831 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:50:25.831 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:50:25.831 INFO:journalctl@ceph.mon.b.smithi170.stdout: Traceback (most recent call last): 2024-05-06T13:50:25.831 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:50:25.832 INFO:journalctl@ceph.mon.b.smithi170.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:50:25.832 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:50:25.832 INFO:journalctl@ceph.mon.b.smithi170.stdout: exec(code, run_globals) 2024-05-06T13:50:25.832 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:50:25.832 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:50:25.832 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:50:25.832 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:50:25.832 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:50:25.832 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:50:25.832 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:50:25.832 INFO:journalctl@ceph.mon.b.smithi170.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:50:25.832 INFO:journalctl@ceph.mon.b.smithi170.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:50:25.833 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:25 smithi170 ceph-mon[23272]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-06T13:50:25.833 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:25 smithi170 ceph-mon[23272]: Reconfiguring daemon prometheus.a on smithi170 2024-05-06T13:50:26.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:25 smithi077 ceph-mon[22428]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:50:26.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:50:26.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:50:26.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:50:26.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:50:26.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:50:26.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:50:26.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:50:26.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:50:26.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:50:26.053 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:50:26.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:50:26.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:50:26.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:50:26.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:50:26.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:50:26.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:50:26.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:50:26.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:50:26.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:50:26.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:50:26.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:50:26.054 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:50:26.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:50:26.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:50:26.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:50:26.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:50:26.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:50:26.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:50:26.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: return future.result(timeout) 2024-05-06T13:50:26.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:50:26.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: return self.__get_result() 2024-05-06T13:50:26.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:50:26.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: raise self._exception 2024-05-06T13:50:26.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:50:26.055 INFO:journalctl@ceph.mon.a.smithi077.stdout: use_current_daemon_image=reconfig, 2024-05-06T13:50:26.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:50:26.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:50:26.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:50:26.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:50:26.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:50:26.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:50:26.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:50:26.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:50:26.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:50:26.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:50:26.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:50:26.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:50:26.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:50:26.056 INFO:journalctl@ceph.mon.a.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:50:26.057 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:50:26.057 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:50:26.057 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:50:26.057 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:50:26.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:25 smithi077 ceph-mon[27166]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:50:26.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:50:26.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:50:26.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:50:26.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:50:26.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:50:26.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:50:26.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:50:26.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:50:26.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:50:26.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:50:26.058 INFO:journalctl@ceph.mon.c.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:50:26.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:50:26.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:50:26.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:50:26.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:50:26.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:50:26.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:50:26.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:50:26.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:50:26.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:50:26.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:50:26.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:50:26.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:50:26.059 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:50:26.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:50:26.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:50:26.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:50:26.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:50:26.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: return future.result(timeout) 2024-05-06T13:50:26.060 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:50:26.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: return self.__get_result() 2024-05-06T13:50:26.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:50:26.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: raise self._exception 2024-05-06T13:50:26.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:50:26.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: use_current_daemon_image=reconfig, 2024-05-06T13:50:26.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:50:26.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:50:26.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:50:26.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:50:26.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:50:26.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:50:26.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:50:26.061 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:50:26.062 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:50:26.062 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:50:26.062 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:50:26.062 INFO:journalctl@ceph.mon.c.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:50:26.062 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:50:26.062 INFO:journalctl@ceph.mon.c.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:50:26.062 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:50:26.062 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:50:26.062 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:50:26.062 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:50:26.063 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:50:26.063 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:50:26.063 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:50:26.063 INFO:journalctl@ceph.mon.a.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:50:26.063 INFO:journalctl@ceph.mon.a.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:50:26.063 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:25 smithi077 ceph-mon[22428]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-06T13:50:26.063 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:25 smithi077 ceph-mon[22428]: Reconfiguring daemon prometheus.a on smithi170 2024-05-06T13:50:26.063 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:50:26.064 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:50:26.064 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:50:26.064 INFO:journalctl@ceph.mon.c.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:50:26.064 INFO:journalctl@ceph.mon.c.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:50:26.064 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:25 smithi077 ceph-mon[27166]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-06T13:50:26.064 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:25 smithi077 ceph-mon[27166]: Reconfiguring daemon prometheus.a on smithi170 2024-05-06T13:50:26.962 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:26 smithi170 ceph-mon[23272]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-06T13:50:27.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:26 smithi077 ceph-mon[22428]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-06T13:50:27.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:26 smithi077 ceph-mon[27166]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-06T13:50:28.077 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:27 smithi170 systemd[1]: Stopping Ceph prometheus.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:50:28.077 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:27 smithi170 bash[55195]: Error: no container with name or ID "ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus.a" found: no such container 2024-05-06T13:50:28.361 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:50:28.106Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-05-06T13:50:28.361 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:50:28.106Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-05-06T13:50:28.362 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:50:28.106Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-05-06T13:50:28.362 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:50:28.106Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-05-06T13:50:28.362 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:50:28.106Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-05-06T13:50:28.362 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:50:28.106Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-05-06T13:50:28.362 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:50:28.106Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-05-06T13:50:28.362 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:50:28.106Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-05-06T13:50:28.362 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:50:28.106Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-05-06T13:50:28.362 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:50:28.107Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-05-06T13:50:28.362 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:50:28.107Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-05-06T13:50:28.362 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[52236]: ts=2024-05-06T13:50:28.107Z caller=main.go:1066 level=info msg="See you next time!" 2024-05-06T13:50:28.362 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:28 smithi170 podman[55208]: 2024-05-06 13:50:28.11677412 +0000 UTC m=+0.127666256 container died 7550480b5a53088a870bfe3f3d9312fd1c1127babe29f46a60b6592852dfe332 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:50:28.652 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:28 smithi170 podman[55208]: 2024-05-06 13:50:28.443125677 +0000 UTC m=+0.454017813 container cleanup 7550480b5a53088a870bfe3f3d9312fd1c1127babe29f46a60b6592852dfe332 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:50:28.652 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:28 smithi170 bash[55208]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a 2024-05-06T13:50:28.653 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:28 smithi170 bash[55236]: Error: no container with name or ID "ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus.a" found: no such container 2024-05-06T13:50:28.937 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:28 smithi170 ceph-mon[23272]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:50:28.937 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:28 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:50:28.938 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:28 smithi170 podman[55222]: 2024-05-06 13:50:28.652622941 +0000 UTC m=+0.534212560 container remove 7550480b5a53088a870bfe3f3d9312fd1c1127babe29f46a60b6592852dfe332 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:50:28.938 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:28 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-05-06T13:50:28.938 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:28 smithi170 systemd[1]: Stopped Ceph prometheus.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:50:28.938 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:28 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@prometheus.a.service: Consumed 1.299s CPU time. 2024-05-06T13:50:28.938 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:28 smithi170 systemd[1]: Starting Ceph prometheus.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:50:28.938 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:28 smithi170 podman[55304]: 2024-05-06 13:50:28.787279098 +0000 UTC m=+0.018989133 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-05-06T13:50:28.938 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:28 smithi170 podman[55304]: 2024-05-06 13:50:28.937640697 +0000 UTC m=+0.169350765 container create 7a5811bc6799e0a4b5031993e175a49d45053bdd299741b4699a7c112eff44f5 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:50:29.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:28 smithi077 ceph-mon[22428]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:50:29.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:28 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:50:29.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:28 smithi077 ceph-mon[27166]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:50:29.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:28 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:50:29.199 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:29 smithi170 podman[55304]: 2024-05-06 13:50:29.17346581 +0000 UTC m=+0.405175835 container init 7a5811bc6799e0a4b5031993e175a49d45053bdd299741b4699a7c112eff44f5 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:50:29.199 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:29 smithi170 podman[55304]: 2024-05-06 13:50:29.176090176 +0000 UTC m=+0.407800202 container start 7a5811bc6799e0a4b5031993e175a49d45053bdd299741b4699a7c112eff44f5 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:50:29.475 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:50:29] ENGINE Bus STOPPING 2024-05-06T13:50:29.475 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:50:29] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-06T13:50:29.475 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:50:29] ENGINE Bus STOPPED 2024-05-06T13:50:29.475 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:50:29] ENGINE Bus STARTING 2024-05-06T13:50:29.476 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:50:29] ENGINE Serving on http://:::9283 2024-05-06T13:50:29.476 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:50:29] ENGINE Bus STARTED 2024-05-06T13:50:29.476 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:50:29.199Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-05-06T13:50:29.476 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:50:29.199Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-05-06T13:50:29.476 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:50:29.199Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-05-06T13:50:29.476 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:50:29.199Z caller=main.go:518 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi170 (none))" 2024-05-06T13:50:29.476 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:50:29.199Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-05-06T13:50:29.476 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:50:29.199Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-05-06T13:50:29.476 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:50:29.201Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-05-06T13:50:29.476 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:50:29.201Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-05-06T13:50:29.476 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:50:29.202Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-05-06T13:50:29.476 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:50:29.204Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-05-06T13:50:29.477 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:50:29.204Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.547µs 2024-05-06T13:50:29.477 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:50:29.205Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-05-06T13:50:29.477 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:29 smithi170 bash[55304]: 7a5811bc6799e0a4b5031993e175a49d45053bdd299741b4699a7c112eff44f5 2024-05-06T13:50:29.477 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:29 smithi170 systemd[1]: Started Ceph prometheus.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:50:29.477 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:50:29.419Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=4 2024-05-06T13:50:29.746 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:50:29.642Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=4 2024-05-06T13:50:29.747 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:50:29.693Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=4 2024-05-06T13:50:30.076 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:50:29.746Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=4 2024-05-06T13:50:30.076 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:50:29.747Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=4 2024-05-06T13:50:30.077 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:50:29.747Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=57.087µs wal_replay_duration=542.057247ms total_replay_duration=542.135803ms 2024-05-06T13:50:30.077 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:50:29.752Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-05-06T13:50:30.077 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:50:29.752Z caller=main.go:947 level=info msg="TSDB started" 2024-05-06T13:50:30.077 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:50:29.752Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-05-06T13:50:30.077 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:50:29.769Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.543591ms db_storage=783ns remote_storage=1.43µs web_handler=411ns query_engine=695ns scrape=726.872µs scrape_sd=97.129µs notify=15.356µs notify_sd=25.982µs rules=16.330747ms 2024-05-06T13:50:30.077 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:50:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:50:29.769Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-05-06T13:50:30.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:30 smithi077 ceph-mon[27166]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:50:30.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:30 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:30.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:30 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:30.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:30 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:50:30.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:30 smithi077 ceph-mon[27166]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:50:30.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:30 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://host.containers.internal:9095"}]: dispatch 2024-05-06T13:50:30.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:30 smithi077 ceph-mon[27166]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://host.containers.internal:9095"}]: dispatch 2024-05-06T13:50:30.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:30 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:30.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:30 smithi077 ceph-mon[22428]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:50:30.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:30 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:30.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:30 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:30.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:30 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:50:30.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:30 smithi077 ceph-mon[22428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:50:30.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:30 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://host.containers.internal:9095"}]: dispatch 2024-05-06T13:50:30.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:30 smithi077 ceph-mon[22428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://host.containers.internal:9095"}]: dispatch 2024-05-06T13:50:30.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:30 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:30.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:30 smithi170 ceph-mon[23272]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:50:30.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:30 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:30.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:30 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:30.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:30 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:50:30.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:30 smithi170 ceph-mon[23272]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:50:30.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:30 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://host.containers.internal:9095"}]: dispatch 2024-05-06T13:50:30.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:30 smithi170 ceph-mon[23272]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://host.containers.internal:9095"}]: dispatch 2024-05-06T13:50:30.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:30 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:32.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:32 smithi170 ceph-mon[23272]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:50:32.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:32 smithi077 ceph-mon[22428]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:50:32.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:32 smithi077 ceph-mon[27166]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:50:35.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:34 smithi077 ceph-mon[22428]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-05-06T13:50:35.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:34 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:35.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:34 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:35.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:34 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:50:35.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:34 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:50:35.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:34 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:35.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:34 smithi077 ceph-mon[22428]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:50:35.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:34 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:50:35.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:34 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:50:35.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:34 smithi077 ceph-mon[27166]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-05-06T13:50:35.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:34 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:35.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:34 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:35.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:34 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:50:35.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:34 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:50:35.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:34 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:35.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:34 smithi077 ceph-mon[27166]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:50:35.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:34 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:50:35.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:34 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:50:35.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:34 smithi170 ceph-mon[23272]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-05-06T13:50:35.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:34 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:35.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:34 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:35.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:34 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:50:35.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:34 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:50:35.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:34 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:35.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:34 smithi170 ceph-mon[23272]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:50:35.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:34 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:50:35.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:34 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:50:35.990 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:35 smithi077 ceph-mon[22428]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:50:35.990 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:35 smithi077 ceph-mon[27166]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:50:36.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:35 smithi170 ceph-mon[23272]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:50:37.022 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:36 smithi077 ceph-mon[22428]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-05-06T13:50:37.022 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:36 smithi077 ceph-mon[27166]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-05-06T13:50:37.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:36 smithi170 ceph-mon[23272]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-05-06T13:50:39.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:38 smithi077 ceph-mon[22428]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:50:39.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:38 smithi077 ceph-mon[27166]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:50:39.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:38 smithi170 ceph-mon[23272]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:50:39.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:39 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:50:39] "GET /metrics HTTP/1.1" 200 37984 "" "Prometheus/2.33.4" 2024-05-06T13:50:41.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:40 smithi077 ceph-mon[22428]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:50:41.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:40 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:41.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:40 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:41.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:40 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:50:41.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:40 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:50:41.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:40 smithi077 ceph-mon[27166]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:50:41.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:40 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:41.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:40 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:41.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:40 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:50:41.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:40 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:50:41.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:40 smithi170 ceph-mon[23272]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:50:41.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:40 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:41.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:40 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:41.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:40 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:50:41.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:40 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:50:42.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:41 smithi077 ceph-mon[22428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:50:42.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:41 smithi077 ceph-mon[22428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:50:42.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:41 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3493031307' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:50:42.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:41 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2814871295' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/665835170"}]: dispatch 2024-05-06T13:50:42.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:41 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/665835170"}]: dispatch 2024-05-06T13:50:42.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:41 smithi077 ceph-mon[27166]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:50:42.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:41 smithi077 ceph-mon[27166]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:50:42.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:41 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3493031307' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:50:42.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:41 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2814871295' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/665835170"}]: dispatch 2024-05-06T13:50:42.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:41 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/665835170"}]: dispatch 2024-05-06T13:50:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:41 smithi170 ceph-mon[23272]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:50:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:41 smithi170 ceph-mon[23272]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:50:42.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:41 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/3493031307' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:50:42.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:41 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2814871295' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/665835170"}]: dispatch 2024-05-06T13:50:42.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:41 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/665835170"}]: dispatch 2024-05-06T13:50:43.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:42 smithi077 ceph-mon[22428]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:50:43.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:42 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/665835170"}]': finished 2024-05-06T13:50:43.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:42 smithi077 ceph-mon[22428]: osdmap e106: 8 total, 8 up, 8 in 2024-05-06T13:50:43.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:42 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/4160901731' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/665835170"}]: dispatch 2024-05-06T13:50:43.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:42 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/665835170"}]: dispatch 2024-05-06T13:50:43.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:42 smithi077 ceph-mon[27166]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:50:43.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:42 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/665835170"}]': finished 2024-05-06T13:50:43.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:42 smithi077 ceph-mon[27166]: osdmap e106: 8 total, 8 up, 8 in 2024-05-06T13:50:43.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:42 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/4160901731' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/665835170"}]: dispatch 2024-05-06T13:50:43.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:42 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/665835170"}]: dispatch 2024-05-06T13:50:43.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:42 smithi170 ceph-mon[23272]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:50:43.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:42 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/665835170"}]': finished 2024-05-06T13:50:43.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:42 smithi170 ceph-mon[23272]: osdmap e106: 8 total, 8 up, 8 in 2024-05-06T13:50:43.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:42 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/4160901731' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/665835170"}]: dispatch 2024-05-06T13:50:43.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:42 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/665835170"}]: dispatch 2024-05-06T13:50:44.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:43 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/665835170"}]': finished 2024-05-06T13:50:44.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:43 smithi077 ceph-mon[22428]: osdmap e107: 8 total, 8 up, 8 in 2024-05-06T13:50:44.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:43 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:50:44.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:43 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/418103127' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2818285713"}]: dispatch 2024-05-06T13:50:44.010 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:43 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/665835170"}]': finished 2024-05-06T13:50:44.010 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:43 smithi077 ceph-mon[27166]: osdmap e107: 8 total, 8 up, 8 in 2024-05-06T13:50:44.010 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:43 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:50:44.010 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:43 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/418103127' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2818285713"}]: dispatch 2024-05-06T13:50:44.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:43 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/665835170"}]': finished 2024-05-06T13:50:44.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:43 smithi170 ceph-mon[23272]: osdmap e107: 8 total, 8 up, 8 in 2024-05-06T13:50:44.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:43 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:50:44.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:43 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/418103127' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2818285713"}]: dispatch 2024-05-06T13:50:45.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:44 smithi077 ceph-mon[22428]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:50:45.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:44 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/418103127' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2818285713"}]': finished 2024-05-06T13:50:45.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:44 smithi077 ceph-mon[22428]: osdmap e108: 8 total, 8 up, 8 in 2024-05-06T13:50:45.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:44 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/419478637' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2192939145"}]: dispatch 2024-05-06T13:50:45.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:44 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2192939145"}]: dispatch 2024-05-06T13:50:45.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:44 smithi077 ceph-mon[27166]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:50:45.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:44 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/418103127' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2818285713"}]': finished 2024-05-06T13:50:45.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:44 smithi077 ceph-mon[27166]: osdmap e108: 8 total, 8 up, 8 in 2024-05-06T13:50:45.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:44 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/419478637' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2192939145"}]: dispatch 2024-05-06T13:50:45.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:44 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2192939145"}]: dispatch 2024-05-06T13:50:45.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:44 smithi170 ceph-mon[23272]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:50:45.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:44 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/418103127' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2818285713"}]': finished 2024-05-06T13:50:45.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:44 smithi170 ceph-mon[23272]: osdmap e108: 8 total, 8 up, 8 in 2024-05-06T13:50:45.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:44 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/419478637' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2192939145"}]: dispatch 2024-05-06T13:50:45.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:44 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2192939145"}]: dispatch 2024-05-06T13:50:46.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:45 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2192939145"}]': finished 2024-05-06T13:50:46.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:45 smithi077 ceph-mon[22428]: osdmap e109: 8 total, 8 up, 8 in 2024-05-06T13:50:46.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:45 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:46.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:45 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:46.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:45 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:50:46.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:45 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:50:46.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:45 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:46.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:45 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/1685756666' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/4087490399"}]: dispatch 2024-05-06T13:50:46.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:45 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/4087490399"}]: dispatch 2024-05-06T13:50:46.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:45 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2192939145"}]': finished 2024-05-06T13:50:46.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:45 smithi077 ceph-mon[27166]: osdmap e109: 8 total, 8 up, 8 in 2024-05-06T13:50:46.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:45 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:46.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:45 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:46.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:45 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:50:46.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:45 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:50:46.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:45 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:46.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:45 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/1685756666' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/4087490399"}]: dispatch 2024-05-06T13:50:46.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:45 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/4087490399"}]: dispatch 2024-05-06T13:50:46.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:45 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2192939145"}]': finished 2024-05-06T13:50:46.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:45 smithi170 ceph-mon[23272]: osdmap e109: 8 total, 8 up, 8 in 2024-05-06T13:50:46.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:45 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:46.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:45 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:46.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:45 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:50:46.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:45 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:50:46.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:45 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:50:46.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:45 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/1685756666' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/4087490399"}]: dispatch 2024-05-06T13:50:46.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:45 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/4087490399"}]: dispatch 2024-05-06T13:50:47.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:46 smithi077 ceph-mon[22428]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-05-06T13:50:47.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:46 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/4087490399"}]': finished 2024-05-06T13:50:47.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:46 smithi077 ceph-mon[22428]: osdmap e110: 8 total, 8 up, 8 in 2024-05-06T13:50:47.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:46 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/1818974965' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1318355975"}]: dispatch 2024-05-06T13:50:47.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:46 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1318355975"}]: dispatch 2024-05-06T13:50:47.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:46 smithi077 ceph-mon[27166]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-05-06T13:50:47.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:46 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/4087490399"}]': finished 2024-05-06T13:50:47.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:46 smithi077 ceph-mon[27166]: osdmap e110: 8 total, 8 up, 8 in 2024-05-06T13:50:47.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:46 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/1818974965' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1318355975"}]: dispatch 2024-05-06T13:50:47.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:46 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1318355975"}]: dispatch 2024-05-06T13:50:47.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:46 smithi170 ceph-mon[23272]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-05-06T13:50:47.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:46 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/4087490399"}]': finished 2024-05-06T13:50:47.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:46 smithi170 ceph-mon[23272]: osdmap e110: 8 total, 8 up, 8 in 2024-05-06T13:50:47.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:46 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/1818974965' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1318355975"}]: dispatch 2024-05-06T13:50:47.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:46 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1318355975"}]: dispatch 2024-05-06T13:50:48.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:47 smithi077 ceph-mon[22428]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 995 B/s rd, 1 op/s 2024-05-06T13:50:48.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:47 smithi077 ceph-mon[22428]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1318355975"}]': finished 2024-05-06T13:50:48.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:47 smithi077 ceph-mon[22428]: osdmap e111: 8 total, 8 up, 8 in 2024-05-06T13:50:48.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:47 smithi077 ceph-mon[27166]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 995 B/s rd, 1 op/s 2024-05-06T13:50:48.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:47 smithi077 ceph-mon[27166]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1318355975"}]': finished 2024-05-06T13:50:48.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:47 smithi077 ceph-mon[27166]: osdmap e111: 8 total, 8 up, 8 in 2024-05-06T13:50:48.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:47 smithi170 ceph-mon[23272]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 995 B/s rd, 1 op/s 2024-05-06T13:50:48.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:47 smithi170 ceph-mon[23272]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/1318355975"}]': finished 2024-05-06T13:50:48.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:47 smithi170 ceph-mon[23272]: osdmap e111: 8 total, 8 up, 8 in 2024-05-06T13:50:49.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:49 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:50:49] "GET /metrics HTTP/1.1" 200 37980 "" "Prometheus/2.33.4" 2024-05-06T13:50:51.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:50 smithi077 ceph-mon[27166]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-05-06T13:50:51.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:50 smithi077 ceph-mon[22428]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-05-06T13:50:51.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:50 smithi170 ceph-mon[23272]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-05-06T13:50:53.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:52 smithi077 ceph-mon[22428]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:50:53.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:52 smithi077 ceph-mon[22428]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 838 B/s rd, 0 op/s 2024-05-06T13:50:53.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:52 smithi077 ceph-mon[27166]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:50:53.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:52 smithi077 ceph-mon[27166]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 838 B/s rd, 0 op/s 2024-05-06T13:50:53.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:52 smithi170 ceph-mon[23272]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:50:53.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:52 smithi170 ceph-mon[23272]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 838 B/s rd, 0 op/s 2024-05-06T13:50:55.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:54 smithi077 ceph-mon[22428]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-05-06T13:50:55.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:54 smithi077 ceph-mon[27166]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-05-06T13:50:55.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:54 smithi170 ceph-mon[23272]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-05-06T13:50:57.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:56 smithi077 ceph-mon[22428]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T13:50:57.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:56 smithi077 ceph-mon[27166]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T13:50:57.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:56 smithi170 ceph-mon[23272]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T13:50:59.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:58 smithi077 ceph-mon[22428]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T13:50:59.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:50:58 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:50:59.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:58 smithi077 ceph-mon[27166]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T13:50:59.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:50:58 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:50:59.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:58 smithi170 ceph-mon[23272]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T13:50:59.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:50:58 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:50:59.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:50:59 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:50:59] "GET /metrics HTTP/1.1" 200 37983 "" "Prometheus/2.33.4" 2024-05-06T13:51:01.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:00 smithi077 ceph-mon[22428]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-06T13:51:01.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:00 smithi077 ceph-mon[27166]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-06T13:51:01.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:00 smithi170 ceph-mon[23272]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-06T13:51:03.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:02 smithi077 ceph-mon[22428]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:51:03.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:02 smithi077 ceph-mon[22428]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:03.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:02 smithi077 ceph-mon[27166]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:51:03.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:02 smithi077 ceph-mon[27166]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:03.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:02 smithi170 ceph-mon[23272]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:51:03.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:02 smithi170 ceph-mon[23272]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:05.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:04 smithi077 ceph-mon[22428]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:05.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:04 smithi077 ceph-mon[27166]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:05.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:04 smithi170 ceph-mon[23272]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:07.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:06 smithi077 ceph-mon[22428]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:07.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:06 smithi077 ceph-mon[27166]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:07.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:06 smithi170 ceph-mon[23272]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:09.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:08 smithi077 ceph-mon[22428]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:51:09.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:08 smithi077 ceph-mon[27166]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:51:09.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:08 smithi170 ceph-mon[23272]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:51:09.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:51:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:51:09] "GET /metrics HTTP/1.1" 200 37983 "" "Prometheus/2.33.4" 2024-05-06T13:51:11.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:10 smithi077 ceph-mon[22428]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:11.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:10 smithi077 ceph-mon[27166]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:11.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:10 smithi170 ceph-mon[23272]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:13.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:12 smithi077 ceph-mon[22428]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:51:13.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:12 smithi077 ceph-mon[22428]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:51:13.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:12 smithi077 ceph-mon[27166]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:51:13.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:12 smithi077 ceph-mon[27166]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:51:13.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:12 smithi170 ceph-mon[23272]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:51:13.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:12 smithi170 ceph-mon[23272]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:51:14.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:13 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:51:14.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:13 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:51:14.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:13 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:51:15.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:14 smithi077 ceph-mon[22428]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:51:15.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:14 smithi077 ceph-mon[27166]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:51:15.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:14 smithi170 ceph-mon[23272]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:51:17.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:16 smithi077 ceph-mon[22428]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:17.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:16 smithi077 ceph-mon[27166]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:17.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:16 smithi170 ceph-mon[23272]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:19.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:18 smithi077 ceph-mon[22428]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:19.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:18 smithi077 ceph-mon[27166]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:19.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:18 smithi170 ceph-mon[23272]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:19.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:51:19 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:51:19] "GET /metrics HTTP/1.1" 200 37981 "" "Prometheus/2.33.4" 2024-05-06T13:51:21.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:20 smithi077 ceph-mon[22428]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:21.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:20 smithi077 ceph-mon[27166]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:21.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:20 smithi170 ceph-mon[23272]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:23.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:22 smithi077 ceph-mon[27166]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:51:23.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:22 smithi077 ceph-mon[27166]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:23.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:22 smithi077 ceph-mon[22428]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:51:23.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:22 smithi077 ceph-mon[22428]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:23.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:22 smithi170 ceph-mon[23272]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:51:23.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:22 smithi170 ceph-mon[23272]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:25.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:24 smithi077 ceph-mon[22428]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:25.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:24 smithi077 ceph-mon[27166]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:25.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:24 smithi170 ceph-mon[23272]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:27.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:26 smithi077 ceph-mon[22428]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:27.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:26 smithi077 ceph-mon[27166]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:27.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:26 smithi170 ceph-mon[23272]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:29.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:28 smithi170 ceph-mon[23272]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:29.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:28 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:51:29.108 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:28 smithi077 ceph-mon[22428]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:29.108 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:28 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:51:29.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:28 smithi077 ceph-mon[27166]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:29.108 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:28 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:51:29.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:51:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:51:29] "GET /metrics HTTP/1.1" 200 37977 "" "Prometheus/2.33.4" 2024-05-06T13:51:31.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:30 smithi170 ceph-mon[23272]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:31.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:30 smithi077 ceph-mon[22428]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:31.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:30 smithi077 ceph-mon[27166]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:33.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:32 smithi170 ceph-mon[23272]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:33.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:32 smithi170 ceph-mon[23272]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:51:33.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:32 smithi077 ceph-mon[22428]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:33.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:32 smithi077 ceph-mon[22428]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:51:33.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:32 smithi077 ceph-mon[27166]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:33.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:32 smithi077 ceph-mon[27166]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:51:35.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:34 smithi170 ceph-mon[23272]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:35.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:34 smithi077 ceph-mon[22428]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:35.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:34 smithi077 ceph-mon[27166]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:37.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:36 smithi170 ceph-mon[23272]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:37.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:36 smithi077 ceph-mon[22428]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:37.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:36 smithi077 ceph-mon[27166]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:39.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:38 smithi170 ceph-mon[23272]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:39.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:38 smithi077 ceph-mon[22428]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:39.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:38 smithi077 ceph-mon[27166]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:39.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:51:39 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:51:39] "GET /metrics HTTP/1.1" 200 37977 "" "Prometheus/2.33.4" 2024-05-06T13:51:41.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:40 smithi170 ceph-mon[23272]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:41.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:40 smithi077 ceph-mon[22428]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:41.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:40 smithi077 ceph-mon[27166]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:43.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:42 smithi170 ceph-mon[23272]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:43.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:42 smithi170 ceph-mon[23272]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:51:43.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:42 smithi077 ceph-mon[22428]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:43.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:42 smithi077 ceph-mon[22428]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:51:43.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:42 smithi077 ceph-mon[27166]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:43.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:42 smithi077 ceph-mon[27166]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:51:44.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:43 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:51:44.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:43 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:51:44.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:43 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:51:45.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:44 smithi077 ceph-mon[22428]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:45.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:44 smithi077 ceph-mon[27166]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:45.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:44 smithi170 ceph-mon[23272]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:46.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:45 smithi077 ceph-mon[22428]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:46.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:45 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:51:46.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:45 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:51:46.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:45 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:51:46.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:45 smithi077 ceph-mon[27166]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:46.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:45 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:51:46.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:45 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:51:46.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:45 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:51:46.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:45 smithi170 ceph-mon[23272]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:46.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:45 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:51:46.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:45 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:51:46.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:45 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:51:48.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:47 smithi077 ceph-mon[22428]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:48.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:47 smithi077 ceph-mon[27166]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:48.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:47 smithi170 ceph-mon[23272]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:49.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:51:49 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:51:49] "GET /metrics HTTP/1.1" 200 37981 "" "Prometheus/2.33.4" 2024-05-06T13:51:51.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:50 smithi077 ceph-mon[22428]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:51.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:50 smithi077 ceph-mon[27166]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:51.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:50 smithi170 ceph-mon[23272]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:53.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:52 smithi077 ceph-mon[22428]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:53.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:52 smithi077 ceph-mon[22428]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:51:53.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:52 smithi077 ceph-mon[27166]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:53.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:52 smithi077 ceph-mon[27166]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:51:53.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:52 smithi170 ceph-mon[23272]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:53.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:52 smithi170 ceph-mon[23272]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:51:55.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:54 smithi077 ceph-mon[22428]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:55.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:54 smithi077 ceph-mon[27166]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:55.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:54 smithi170 ceph-mon[23272]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:57.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:56 smithi077 ceph-mon[22428]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:57.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:56 smithi077 ceph-mon[27166]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:57.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:56 smithi170 ceph-mon[23272]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:51:59.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:58 smithi077 ceph-mon[22428]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:59.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:51:58 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:51:59.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:58 smithi077 ceph-mon[27166]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:59.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:51:58 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:51:59.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:58 smithi170 ceph-mon[23272]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:51:59.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:51:58 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:51:59.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:51:59 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:51:59] "GET /metrics HTTP/1.1" 200 37983 "" "Prometheus/2.33.4" 2024-05-06T13:52:01.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:00 smithi077 ceph-mon[22428]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:01.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:00 smithi077 ceph-mon[27166]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:01.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:00 smithi170 ceph-mon[23272]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:03.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:02 smithi077 ceph-mon[22428]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:03.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:02 smithi077 ceph-mon[22428]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:52:03.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:02 smithi077 ceph-mon[27166]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:03.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:02 smithi077 ceph-mon[27166]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:52:03.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:02 smithi170 ceph-mon[23272]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:03.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:02 smithi170 ceph-mon[23272]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:52:05.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:04 smithi077 ceph-mon[22428]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:05.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:04 smithi077 ceph-mon[27166]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:05.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:04 smithi170 ceph-mon[23272]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:07.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:06 smithi077 ceph-mon[22428]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:07.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:06 smithi077 ceph-mon[27166]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:07.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:06 smithi170 ceph-mon[23272]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:09.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:08 smithi077 ceph-mon[22428]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:09.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:08 smithi077 ceph-mon[27166]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:09.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:08 smithi170 ceph-mon[23272]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:09.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:52:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:52:09] "GET /metrics HTTP/1.1" 200 37983 "" "Prometheus/2.33.4" 2024-05-06T13:52:11.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:10 smithi077 ceph-mon[22428]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:11.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:10 smithi077 ceph-mon[27166]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:11.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:10 smithi170 ceph-mon[23272]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:13.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:12 smithi077 ceph-mon[22428]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:13.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:12 smithi077 ceph-mon[22428]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:52:13.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:12 smithi077 ceph-mon[27166]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:13.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:12 smithi077 ceph-mon[27166]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:52:13.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:12 smithi170 ceph-mon[23272]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:13.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:12 smithi170 ceph-mon[23272]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:52:14.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:13 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:52:14.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:13 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:52:14.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:13 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:52:15.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:14 smithi077 ceph-mon[22428]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:15.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:14 smithi077 ceph-mon[27166]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:15.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:14 smithi170 ceph-mon[23272]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:17.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:16 smithi077 ceph-mon[22428]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:17.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:16 smithi077 ceph-mon[27166]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:17.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:16 smithi170 ceph-mon[23272]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:19.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:18 smithi077 ceph-mon[22428]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:19.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:18 smithi077 ceph-mon[27166]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:19.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:18 smithi170 ceph-mon[23272]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:19.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:52:19 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:52:19] "GET /metrics HTTP/1.1" 200 37981 "" "Prometheus/2.33.4" 2024-05-06T13:52:21.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:20 smithi077 ceph-mon[22428]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:21.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:20 smithi077 ceph-mon[27166]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:21.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:20 smithi170 ceph-mon[23272]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:23.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:22 smithi170 ceph-mon[23272]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:23.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:22 smithi170 ceph-mon[23272]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:52:23.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:22 smithi077 ceph-mon[22428]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:23.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:22 smithi077 ceph-mon[22428]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:52:23.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:22 smithi077 ceph-mon[27166]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:23.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:22 smithi077 ceph-mon[27166]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:52:25.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:24 smithi170 ceph-mon[23272]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:25.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:24 smithi077 ceph-mon[22428]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:25.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:24 smithi077 ceph-mon[27166]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:27.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:26 smithi170 ceph-mon[23272]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:27.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:26 smithi077 ceph-mon[22428]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:27.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:26 smithi077 ceph-mon[27166]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:29.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:28 smithi170 ceph-mon[23272]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:29.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:28 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:52:29.105 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:28 smithi077 ceph-mon[22428]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:29.105 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:28 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:52:29.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:28 smithi077 ceph-mon[27166]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:29.105 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:28 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:52:29.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:52:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:52:29] "GET /metrics HTTP/1.1" 200 37993 "" "Prometheus/2.33.4" 2024-05-06T13:52:31.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:30 smithi170 ceph-mon[23272]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:31.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:30 smithi077 ceph-mon[22428]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:31.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:30 smithi077 ceph-mon[27166]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:33.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:32 smithi170 ceph-mon[23272]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:33.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:32 smithi170 ceph-mon[23272]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:52:33.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:32 smithi077 ceph-mon[22428]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:33.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:32 smithi077 ceph-mon[22428]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:52:33.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:32 smithi077 ceph-mon[27166]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:33.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:32 smithi077 ceph-mon[27166]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:52:35.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:34 smithi170 ceph-mon[23272]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:35.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:34 smithi077 ceph-mon[22428]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:35.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:34 smithi077 ceph-mon[27166]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:37.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:36 smithi170 ceph-mon[23272]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:37.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:36 smithi077 ceph-mon[22428]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:37.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:36 smithi077 ceph-mon[27166]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:39.168 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:38 smithi170 ceph-mon[23272]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:39.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:38 smithi077 ceph-mon[22428]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:39.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:38 smithi077 ceph-mon[27166]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:39.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:52:39 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:52:39] "GET /metrics HTTP/1.1" 200 37993 "" "Prometheus/2.33.4" 2024-05-06T13:52:41.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:40 smithi077 ceph-mon[22428]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:41.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:40 smithi077 ceph-mon[27166]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:41.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:40 smithi170 ceph-mon[23272]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:43.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:42 smithi077 ceph-mon[22428]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:43.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:42 smithi077 ceph-mon[22428]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:52:43.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:42 smithi077 ceph-mon[27166]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:43.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:42 smithi077 ceph-mon[27166]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:52:43.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:42 smithi170 ceph-mon[23272]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:43.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:42 smithi170 ceph-mon[23272]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:52:44.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:43 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:52:44.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:43 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:52:44.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:43 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:52:45.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:44 smithi077 ceph-mon[22428]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:45.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:44 smithi077 ceph-mon[27166]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:45.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:44 smithi170 ceph-mon[23272]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:46.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:45 smithi077 ceph-mon[22428]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:46.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:45 smithi077 ceph-mon[27166]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:46.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:45 smithi170 ceph-mon[23272]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:47.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:46 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:52:47.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:46 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:52:47.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:46 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:52:47.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:46 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:52:47.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:46 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:52:47.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:46 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:52:47.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:46 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:52:47.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:46 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:52:47.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:46 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:52:48.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:47 smithi077 ceph-mon[22428]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:48.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:47 smithi077 ceph-mon[27166]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:48.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:47 smithi170 ceph-mon[23272]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:49.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:52:49 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:52:49] "GET /metrics HTTP/1.1" 200 37993 "" "Prometheus/2.33.4" 2024-05-06T13:52:51.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:50 smithi077 ceph-mon[22428]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:51.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:50 smithi077 ceph-mon[27166]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:51.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:50 smithi170 ceph-mon[23272]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:53.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:52 smithi077 ceph-mon[22428]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:53.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:52 smithi077 ceph-mon[22428]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:52:53.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:52 smithi077 ceph-mon[27166]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:53.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:52 smithi077 ceph-mon[27166]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:52:53.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:52 smithi170 ceph-mon[23272]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:53.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:52 smithi170 ceph-mon[23272]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:52:55.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:54 smithi077 ceph-mon[22428]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:55.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:54 smithi077 ceph-mon[27166]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:55.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:54 smithi170 ceph-mon[23272]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:57.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:56 smithi077 ceph-mon[22428]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:57.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:56 smithi077 ceph-mon[27166]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:57.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:56 smithi170 ceph-mon[23272]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:52:59.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:58 smithi077 ceph-mon[22428]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:59.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:52:58 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:52:59.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:58 smithi077 ceph-mon[27166]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:59.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:52:58 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:52:59.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:58 smithi170 ceph-mon[23272]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:52:59.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:52:58 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:52:59.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:52:59 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:52:59] "GET /metrics HTTP/1.1" 200 37993 "" "Prometheus/2.33.4" 2024-05-06T13:53:01.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:00 smithi077 ceph-mon[22428]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:53:01.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:00 smithi077 ceph-mon[27166]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:53:01.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:00 smithi170 ceph-mon[23272]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:53:03.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:02 smithi077 ceph-mon[22428]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:03.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:02 smithi077 ceph-mon[22428]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:53:03.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:02 smithi077 ceph-mon[27166]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:03.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:02 smithi077 ceph-mon[27166]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:53:03.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:02 smithi170 ceph-mon[23272]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:03.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:02 smithi170 ceph-mon[23272]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:53:05.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:04 smithi077 ceph-mon[22428]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:05.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:04 smithi077 ceph-mon[27166]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:05.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:04 smithi170 ceph-mon[23272]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:07.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:06 smithi077 ceph-mon[22428]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:53:07.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:06 smithi077 ceph-mon[27166]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:53:07.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:06 smithi170 ceph-mon[23272]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:53:09.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:08 smithi077 ceph-mon[22428]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:09.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:08 smithi077 ceph-mon[27166]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:09.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:08 smithi170 ceph-mon[23272]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:09.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:53:09] "GET /metrics HTTP/1.1" 200 37993 "" "Prometheus/2.33.4" 2024-05-06T13:53:11.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:10 smithi077 ceph-mon[22428]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:53:11.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:10 smithi077 ceph-mon[27166]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:53:11.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:10 smithi170 ceph-mon[23272]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:53:13.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:12 smithi077 ceph-mon[22428]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:13.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:12 smithi077 ceph-mon[22428]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:53:13.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:12 smithi077 ceph-mon[27166]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:13.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:12 smithi077 ceph-mon[27166]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:53:13.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:12 smithi170 ceph-mon[23272]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:13.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:12 smithi170 ceph-mon[23272]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:53:14.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:13 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:53:14.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:13 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:53:14.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:13 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:53:15.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:14 smithi170 ceph-mon[23272]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:15.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:14 smithi077 ceph-mon[22428]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:15.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:14 smithi077 ceph-mon[27166]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:16.708 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch ps' 2024-05-06T13:53:17.012 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:16 smithi077 ceph-mon[22428]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:53:17.012 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:16 smithi077 ceph-mon[27166]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:53:17.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:16 smithi170 ceph-mon[23272]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:53:18.652 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T13:53:18.652 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (12m) 2m ago 20m 23.3M - 0.23.0 ba2b418f427c 0f855548777f 2024-05-06T13:53:18.652 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (12m) 2m ago 20m 53.4M - 8.3.5 dad864ee21e9 fdcecac79925 2024-05-06T13:53:18.652 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (2m) 2m ago 19m 50.5M - 3.5 e1d6a67b021e 79b44fb83b69 2024-05-06T13:53:18.652 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283 running (15m) 2m ago 24m 494M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 63ef6825ff68 2024-05-06T13:53:18.652 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (9m) 2m ago 26m 450M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T13:53:18.653 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (26m) 2m ago 26m 68.9M 2048M 17.2.0 e1d6a67b021e ac46f91694f8 2024-05-06T13:53:18.653 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (25m) 2m ago 25m 50.6M 2048M 17.2.0 e1d6a67b021e 24be6fce9d28 2024-05-06T13:53:18.653 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (25m) 2m ago 25m 50.7M 2048M 17.2.0 e1d6a67b021e 83af336ef762 2024-05-06T13:53:18.653 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (12m) 2m ago 21m 17.7M - 1.3.1 1dbe0e931976 54aee5b22b1c 2024-05-06T13:53:18.653 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (12m) 2m ago 21m 18.0M - 1.3.1 1dbe0e931976 ca5df43a7469 2024-05-06T13:53:18.653 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (24m) 2m ago 24m 52.9M 2992M 17.2.0 e1d6a67b021e 2ccee2bb6e96 2024-05-06T13:53:18.653 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (24m) 2m ago 24m 57.1M 2992M 17.2.0 e1d6a67b021e 1d9e6609b585 2024-05-06T13:53:18.653 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (23m) 2m ago 23m 51.9M 2992M 17.2.0 e1d6a67b021e 37a61d295111 2024-05-06T13:53:18.653 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (23m) 2m ago 23m 51.0M 2992M 17.2.0 e1d6a67b021e 8feb7c077aaa 2024-05-06T13:53:18.653 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (23m) 2m ago 23m 55.2M 3248M 17.2.0 e1d6a67b021e f865acc44a06 2024-05-06T13:53:18.653 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (22m) 2m ago 22m 51.2M 3248M 17.2.0 e1d6a67b021e a55bd4122f6f 2024-05-06T13:53:18.653 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (22m) 2m ago 22m 50.4M 3248M 17.2.0 e1d6a67b021e 362938a0b357 2024-05-06T13:53:18.653 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (21m) 2m ago 21m 52.7M 3248M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T13:53:18.653 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (2m) 2m ago 21m 47.6M - 2.33.4 514e6a882f6e 7a5811bc6799 2024-05-06T13:53:18.654 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (19m) 2m ago 19m 76.2M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T13:53:18.654 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (19m) 2m ago 19m 74.7M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T13:53:19.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:18 smithi170 ceph-mon[23272]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:19.185 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:18 smithi077 ceph-mon[22428]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:19.186 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:18 smithi077 ceph-mon[27166]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:19.304 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch ls' 2024-05-06T13:53:19.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:19 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:53:19] "GET /metrics HTTP/1.1" 200 37990 "" "Prometheus/2.33.4" 2024-05-06T13:53:20.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:19 smithi170 ceph-mon[23272]: from='client.15507 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:53:20.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:19 smithi077 ceph-mon[22428]: from='client.15507 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:53:20.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:19 smithi077 ceph-mon[27166]: from='client.15507 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:53:20.842 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:20 smithi077 ceph-mon[22428]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:53:20.843 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:20 smithi077 ceph-mon[27166]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:53:21.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:20 smithi170 ceph-mon[23272]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:53:21.255 INFO:teuthology.orchestra.run.smithi077.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-05-06T13:53:21.256 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager ?:9093,9094 1/1 2m ago 21m smithi077=a;count:1 2024-05-06T13:53:21.256 INFO:teuthology.orchestra.run.smithi077.stdout:grafana ?:3000 1/1 2m ago 21m smithi170=a;count:1 2024-05-06T13:53:21.256 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo ?:5000 1/1 2m ago 19m count:1 2024-05-06T13:53:21.256 INFO:teuthology.orchestra.run.smithi077.stdout:mgr 2/2 2m ago 25m smithi077=y;smithi170=x;count:2 2024-05-06T13:53:21.256 INFO:teuthology.orchestra.run.smithi077.stdout:mon 3/3 2m ago 25m smithi077:172.21.15.77=a;smithi077:[v2:172.21.15.77:3301,v1:172.21.15.77:6790]=c;smithi170:172.21.15.170=b;count:3 2024-05-06T13:53:21.256 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter ?:9100 2/2 2m ago 21m smithi077=a;smithi170=b;count:2 2024-05-06T13:53:21.256 INFO:teuthology.orchestra.run.smithi077.stdout:osd 8 2m ago - 2024-05-06T13:53:21.256 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus ?:9095 1/1 2m ago 21m smithi170=a;count:1 2024-05-06T13:53:21.256 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo ?:8000 2/2 2m ago 19m count:2 2024-05-06T13:53:21.906 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph versions' 2024-05-06T13:53:23.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:22 smithi170 ceph-mon[23272]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:23.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:22 smithi170 ceph-mon[23272]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:53:23.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:22 smithi170 ceph-mon[23272]: from='client.15513 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:53:23.231 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:22 smithi077 ceph-mon[22428]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:23.231 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:22 smithi077 ceph-mon[22428]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:53:23.231 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:22 smithi077 ceph-mon[22428]: from='client.15513 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:53:23.231 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:22 smithi077 ceph-mon[27166]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:23.232 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:22 smithi077 ceph-mon[27166]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:53:23.232 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:22 smithi077 ceph-mon[27166]: from='client.15513 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:53:24.009 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:53:24.009 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-05-06T13:53:24.009 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-05-06T13:53:24.010 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:53:24.010 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-05-06T13:53:24.010 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T13:53:24.010 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:53:24.010 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-05-06T13:53:24.010 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-06T13:53:24.010 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:53:24.010 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": {}, 2024-05-06T13:53:24.010 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw": { 2024-05-06T13:53:24.010 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T13:53:24.010 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:53:24.010 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-05-06T13:53:24.010 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-05-06T13:53:24.010 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T13:53:24.011 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T13:53:24.011 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:53:24.693 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-05-06T13:53:24.999 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:24 smithi077 ceph-mon[22428]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:24.999 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:24 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/878794317' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:53:24.999 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:24 smithi077 ceph-mon[27166]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:24.999 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:24 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/878794317' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:53:25.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:24 smithi170 ceph-mon[23272]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:25.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:24 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/878794317' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:53:27.290 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:26 smithi077 ceph-mon[22428]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:53:27.291 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:26 smithi077 ceph-mon[27166]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:53:27.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:26 smithi170 ceph-mon[23272]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:53:28.271 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:27 smithi077 ceph-mon[22428]: from='client.25429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:53:28.271 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:27 smithi077 ceph-mon[27166]: from='client.25429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:53:28.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:27 smithi170 ceph-mon[23272]: from='client.25429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:53:28.338 INFO:teuthology.orchestra.run.smithi077.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:53:28.933 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:28 smithi077 ceph-mon[22428]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:28.933 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:28 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:53:28.933 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:28 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:53:28.933 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:28 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:53:28.933 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:28 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:53:28.934 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:28 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:53:28.934 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:28 smithi077 ceph-mon[27166]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:28.934 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:28 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:53:28.934 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:28 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:53:28.934 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:28 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:53:28.934 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:28 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:53:28.934 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:28 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:53:28.959 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-05-06T13:53:29.168 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:28 smithi170 ceph-mon[23272]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:29.168 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:28 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:53:29.168 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:28 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:53:29.168 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:28 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:53:29.168 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:28 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:53:29.168 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:28 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:53:29.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ::ffff:172.21.15.170 - - [06/May/2024:13:53:29] "GET /metrics HTTP/1.1" 200 37989 "" "Prometheus/2.33.4" 2024-05-06T13:53:30.158 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:29 smithi077 ceph-mon[22428]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:53:30.159 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:29 smithi077 ceph-mon[22428]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:53:30.159 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:29 smithi077 ceph-mon[27166]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:53:30.159 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:29 smithi077 ceph-mon[27166]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:53:30.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:29 smithi170 ceph-mon[23272]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:53:30.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:29 smithi170 ceph-mon[23272]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:53:31.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:30 smithi077 ceph-mon[27166]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:53:31.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:30 smithi077 ceph-mon[22428]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:53:31.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:30 smithi170 ceph-mon[23272]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:53:31.572 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T13:53:32.224 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T13:53:32.225 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (12m) 2m ago 20m 23.3M - 0.23.0 ba2b418f427c 0f855548777f 2024-05-06T13:53:32.225 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (12m) 2m ago 20m 53.4M - 8.3.5 dad864ee21e9 fdcecac79925 2024-05-06T13:53:32.225 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (2m) 2m ago 19m 50.5M - 3.5 e1d6a67b021e 79b44fb83b69 2024-05-06T13:53:32.225 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283 running (15m) 2m ago 25m 494M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 63ef6825ff68 2024-05-06T13:53:32.225 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (9m) 2m ago 26m 450M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T13:53:32.225 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (26m) 2m ago 26m 68.9M 2048M 17.2.0 e1d6a67b021e ac46f91694f8 2024-05-06T13:53:32.225 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (25m) 2m ago 25m 50.6M 2048M 17.2.0 e1d6a67b021e 24be6fce9d28 2024-05-06T13:53:32.225 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (25m) 2m ago 25m 50.7M 2048M 17.2.0 e1d6a67b021e 83af336ef762 2024-05-06T13:53:32.225 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (12m) 2m ago 21m 17.7M - 1.3.1 1dbe0e931976 54aee5b22b1c 2024-05-06T13:53:32.225 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (12m) 2m ago 21m 18.0M - 1.3.1 1dbe0e931976 ca5df43a7469 2024-05-06T13:53:32.225 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (24m) 2m ago 24m 52.9M 2992M 17.2.0 e1d6a67b021e 2ccee2bb6e96 2024-05-06T13:53:32.225 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (24m) 2m ago 24m 57.1M 2992M 17.2.0 e1d6a67b021e 1d9e6609b585 2024-05-06T13:53:32.225 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (24m) 2m ago 24m 51.9M 2992M 17.2.0 e1d6a67b021e 37a61d295111 2024-05-06T13:53:32.226 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (23m) 2m ago 23m 51.0M 2992M 17.2.0 e1d6a67b021e 8feb7c077aaa 2024-05-06T13:53:32.226 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (23m) 2m ago 23m 55.2M 3248M 17.2.0 e1d6a67b021e f865acc44a06 2024-05-06T13:53:32.226 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (22m) 2m ago 22m 51.2M 3248M 17.2.0 e1d6a67b021e a55bd4122f6f 2024-05-06T13:53:32.226 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (22m) 2m ago 22m 50.4M 3248M 17.2.0 e1d6a67b021e 362938a0b357 2024-05-06T13:53:32.226 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (22m) 2m ago 22m 52.7M 3248M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T13:53:32.226 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (3m) 2m ago 21m 47.6M - 2.33.4 514e6a882f6e 7a5811bc6799 2024-05-06T13:53:32.226 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (19m) 2m ago 19m 76.2M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T13:53:32.226 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (19m) 2m ago 19m 74.7M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T13:53:32.614 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:53:32.614 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-05-06T13:53:32.614 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-05-06T13:53:32.614 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:53:32.614 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-05-06T13:53:32.614 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T13:53:32.614 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:53:32.615 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-05-06T13:53:32.615 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-06T13:53:32.615 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:53:32.615 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": {}, 2024-05-06T13:53:32.615 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw": { 2024-05-06T13:53:32.615 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T13:53:32.615 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:53:32.615 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-05-06T13:53:32.615 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-05-06T13:53:32.615 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T13:53:32.615 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T13:53:32.615 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:53:32.883 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:32 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:53:32] ENGINE Bus STOPPING 2024-05-06T13:53:32.883 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:32 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:53:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-06T13:53:32.883 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:32 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:53:32] ENGINE Bus STOPPED 2024-05-06T13:53:32.883 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:32 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:53:32] ENGINE Bus STARTING 2024-05-06T13:53:32.883 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:32 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:53:32] ENGINE Serving on http://:::9283 2024-05-06T13:53:32.884 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:32 smithi077 ceph-mon[22428]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:32.884 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:32 smithi077 ceph-mon[22428]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:53:32.884 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:32 smithi077 ceph-mon[22428]: from='client.15531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:53:32.884 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:32 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:53:32.884 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:32 smithi077 ceph-mon[22428]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (squid) 2024-05-06T13:53:32.884 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:32 smithi077 ceph-mon[22428]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T13:53:32.884 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:32 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:53:32.884 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:32 smithi077 ceph-mon[22428]: Upgrade: Need to upgrade myself (mgr.x) 2024-05-06T13:53:32.884 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:32 smithi077 ceph-mon[22428]: Upgrade: Need to upgrade myself (mgr.x) 2024-05-06T13:53:32.884 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:32 smithi077 ceph-mon[22428]: Failing over to other MGR 2024-05-06T13:53:32.884 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:32 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-05-06T13:53:32.884 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:32 smithi077 ceph-mon[22428]: osdmap e112: 8 total, 8 up, 8 in 2024-05-06T13:53:32.884 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:32 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/3252354645' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:53:32.885 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:32 smithi077 ceph-mon[27166]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:32.885 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:32 smithi077 ceph-mon[27166]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:53:32.885 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:32 smithi077 ceph-mon[27166]: from='client.15531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:53:32.885 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:32 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:53:32.885 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:32 smithi077 ceph-mon[27166]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (squid) 2024-05-06T13:53:32.885 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:32 smithi077 ceph-mon[27166]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T13:53:32.885 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:32 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:53:32.885 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:32 smithi077 ceph-mon[27166]: Upgrade: Need to upgrade myself (mgr.x) 2024-05-06T13:53:32.885 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:32 smithi077 ceph-mon[27166]: Upgrade: Need to upgrade myself (mgr.x) 2024-05-06T13:53:32.885 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:32 smithi077 ceph-mon[27166]: Failing over to other MGR 2024-05-06T13:53:32.885 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:32 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-05-06T13:53:32.885 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:32 smithi077 ceph-mon[27166]: osdmap e112: 8 total, 8 up, 8 in 2024-05-06T13:53:32.886 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:32 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/3252354645' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:53:32.918 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:32 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:53:32.675+0000 7f5702935700 -1 mgr handle_mgr_map I was active but no longer am 2024-05-06T13:53:32.918 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:32 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ignoring --setuser ceph since I am not root 2024-05-06T13:53:32.918 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:32 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: ignoring --setgroup ceph since I am not root 2024-05-06T13:53:32.918 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:32 smithi170 ceph-mgr[43056]: -- 172.21.15.170:0/1375521313 <== mon.2 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5569cfbbc5a0 con 0x5569cfb91000 2024-05-06T13:53:32.918 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:32 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:53:32.856+0000 7fd8f303d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T13:53:32.919 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:32 smithi170 ceph-mon[23272]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:53:32.919 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:32 smithi170 ceph-mon[23272]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:53:32.919 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:32 smithi170 ceph-mon[23272]: from='client.15531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:53:32.919 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:32 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' 2024-05-06T13:53:32.919 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:32 smithi170 ceph-mon[23272]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (squid) 2024-05-06T13:53:32.919 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:32 smithi170 ceph-mon[23272]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T13:53:32.919 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:32 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:53:32.919 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:32 smithi170 ceph-mon[23272]: Upgrade: Need to upgrade myself (mgr.x) 2024-05-06T13:53:32.919 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:32 smithi170 ceph-mon[23272]: Upgrade: Need to upgrade myself (mgr.x) 2024-05-06T13:53:32.919 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:32 smithi170 ceph-mon[23272]: Failing over to other MGR 2024-05-06T13:53:32.919 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:32 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-05-06T13:53:32.919 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:32 smithi170 ceph-mon[23272]: osdmap e112: 8 total, 8 up, 8 in 2024-05-06T13:53:32.919 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:32 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/3252354645' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:53:33.302 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:32 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:53:32] ENGINE Bus STARTED 2024-05-06T13:53:33.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:33 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:53:33.044+0000 7fd8f303d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:53:33.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:33 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:53:33.148+0000 7fd8f303d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T13:53:33.700 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:53:33.700 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T13:53:33.700 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-05-06T13:53:33.700 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-05-06T13:53:33.700 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [ 2024-05-06T13:53:33.700 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr" 2024-05-06T13:53:33.700 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-05-06T13:53:33.701 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "2/2 daemons upgraded", 2024-05-06T13:53:33.701 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "", 2024-05-06T13:53:33.701 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-05-06T13:53:33.701 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:53:33.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:33 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:53:33.475+0000 7fd8f303d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T13:53:33.827 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:33 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:53:33.564+0000 7fd8f303d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T13:53:33.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:33 smithi170 ceph-mon[23272]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-05-06T13:53:33.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:33 smithi170 ceph-mon[23272]: mgrmap e37: y(active, starting, since 1.00446s) 2024-05-06T13:53:33.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:33 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:53:33.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:33 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:53:33.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:33 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:53:33.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:33 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T13:53:33.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:33 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:53:33.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:33 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:53:33.829 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:33 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:53:33.829 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:33 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:53:33.829 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:33 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:53:33.829 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:33 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:53:33.829 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:33 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:53:33.829 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:33 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:53:33.829 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:33 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T13:53:33.829 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:33 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T13:53:33.830 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:33 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T13:53:33.830 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:33 smithi170 ceph-mon[23272]: Manager daemon y is now available 2024-05-06T13:53:33.830 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:33 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:53:33.830 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:33 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:53:33.830 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:33 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:53:34.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[27166]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-05-06T13:53:34.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[27166]: mgrmap e37: y(active, starting, since 1.00446s) 2024-05-06T13:53:34.009 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:53:34.009 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:53:34.009 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:53:34.009 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T13:53:34.009 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:53:34.009 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:53:34.009 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:53:34.009 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:53:34.009 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:53:34.009 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:53:34.009 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:53:34.009 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:53:34.009 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T13:53:34.009 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T13:53:34.010 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T13:53:34.010 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[27166]: Manager daemon y is now available 2024-05-06T13:53:34.010 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:53:34.010 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:53:34.010 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:53:34.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[22428]: from='mgr.34994 172.21.15.170:0/3480563896' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-05-06T13:53:34.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[22428]: mgrmap e37: y(active, starting, since 1.00446s) 2024-05-06T13:53:34.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:53:34.010 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:53:34.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:53:34.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T13:53:34.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:53:34.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:53:34.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:53:34.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:53:34.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:53:34.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:53:34.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:53:34.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:53:34.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T13:53:34.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T13:53:34.012 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T13:53:34.012 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[22428]: Manager daemon y is now available 2024-05-06T13:53:34.012 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:53:34.012 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:53:34.012 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:33 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:53:34.944 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:34 smithi077 ceph-mon[22428]: mgrmap e38: y(active, since 2s) 2024-05-06T13:53:34.944 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:34 smithi077 ceph-mon[22428]: [06/May/2024:13:53:33] ENGINE Bus STARTING 2024-05-06T13:53:34.944 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:34 smithi077 ceph-mon[22428]: [06/May/2024:13:53:33] ENGINE Serving on https://172.21.15.77:7150 2024-05-06T13:53:34.944 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:34 smithi077 ceph-mon[27166]: mgrmap e38: y(active, since 2s) 2024-05-06T13:53:34.944 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:34 smithi077 ceph-mon[27166]: [06/May/2024:13:53:33] ENGINE Bus STARTING 2024-05-06T13:53:34.944 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:34 smithi077 ceph-mon[27166]: [06/May/2024:13:53:33] ENGINE Serving on https://172.21.15.77:7150 2024-05-06T13:53:35.013 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:34 smithi170 ceph-mon[23272]: mgrmap e38: y(active, since 2s) 2024-05-06T13:53:35.013 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:34 smithi170 ceph-mon[23272]: [06/May/2024:13:53:33] ENGINE Bus STARTING 2024-05-06T13:53:35.013 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:34 smithi170 ceph-mon[23272]: [06/May/2024:13:53:33] ENGINE Serving on https://172.21.15.77:7150 2024-05-06T13:53:35.014 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:34 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:53:34.838+0000 7fd8f303d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T13:53:35.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:53:35.012+0000 7fd8f303d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T13:53:35.327 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:53:35.193+0000 7fd8f303d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T13:53:35.771 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:53:35.508+0000 7fd8f303d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T13:53:35.772 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:53:35.609+0000 7fd8f303d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T13:53:35.772 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:35 smithi170 ceph-mon[23272]: [06/May/2024:13:53:33] ENGINE Serving on http://172.21.15.77:8765 2024-05-06T13:53:35.772 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:35 smithi170 ceph-mon[23272]: [06/May/2024:13:53:33] ENGINE Bus STARTED 2024-05-06T13:53:35.772 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:35 smithi170 ceph-mon[23272]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:53:35.772 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:35 smithi170 ceph-mon[23272]: mgrmap e39: y(active, since 3s) 2024-05-06T13:53:35.930 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:35 smithi077 ceph-mon[22428]: [06/May/2024:13:53:33] ENGINE Serving on http://172.21.15.77:8765 2024-05-06T13:53:35.930 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:35 smithi077 ceph-mon[22428]: [06/May/2024:13:53:33] ENGINE Bus STARTED 2024-05-06T13:53:35.930 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:35 smithi077 ceph-mon[22428]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:53:35.931 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:35 smithi077 ceph-mon[22428]: mgrmap e39: y(active, since 3s) 2024-05-06T13:53:35.931 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:35 smithi077 ceph-mon[27166]: [06/May/2024:13:53:33] ENGINE Serving on http://172.21.15.77:8765 2024-05-06T13:53:35.931 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:35 smithi077 ceph-mon[27166]: [06/May/2024:13:53:33] ENGINE Bus STARTED 2024-05-06T13:53:35.931 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:35 smithi077 ceph-mon[27166]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:53:35.931 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:35 smithi077 ceph-mon[27166]: mgrmap e39: y(active, since 3s) 2024-05-06T13:53:36.076 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:53:35.819+0000 7fd8f303d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T13:53:36.076 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:36 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:53:36.015+0000 7fd8f303d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:53:37.077 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:36 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:53:36.651+0000 7fd8f303d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T13:53:37.471 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:37 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:53:37.163+0000 7fd8f303d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T13:53:37.471 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:37 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:53:37.264+0000 7fd8f303d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T13:53:37.826 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:37 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:53:37.470+0000 7fd8f303d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-06T13:53:37.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:37 smithi170 ceph-mon[23272]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:53:37.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:37 smithi170 ceph-mon[23272]: mgrmap e40: y(active, since 5s) 2024-05-06T13:53:37.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:37 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:37.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:37 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:37.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:37 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:37.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:37 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:38.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:37 smithi077 ceph-mon[27166]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:53:38.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:37 smithi077 ceph-mon[27166]: mgrmap e40: y(active, since 5s) 2024-05-06T13:53:38.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:37 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:38.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:37 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:38.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:37 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:38.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:37 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:38.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:37 smithi077 ceph-mon[22428]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:53:38.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:37 smithi077 ceph-mon[22428]: mgrmap e40: y(active, since 5s) 2024-05-06T13:53:38.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:37 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:38.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:37 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:38.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:37 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:38.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:37 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:38.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:53:38.031+0000 7fd8f303d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T13:53:38.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:53:38.133+0000 7fd8f303d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T13:53:38.770 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:53:38.393+0000 7fd8f303d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T13:53:38.770 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:53:38.537+0000 7fd8f303d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T13:53:38.770 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:53:38.641+0000 7fd8f303d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T13:53:38.771 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:53:38.769+0000 7fd8f303d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T13:53:39.076 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:53:38.876+0000 7fd8f303d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T13:53:40.045 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:39 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:53:39.817+0000 7fd8f303d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T13:53:40.045 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:40 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:53:40.044+0000 7fd8f303d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T13:53:40.045 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:39 smithi170 ceph-mon[23272]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:53:40.045 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:39 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:40.045 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:39 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:40.045 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:39 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:40.045 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:39 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:40.046 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:39 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:53:40.046 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:39 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:53:40.046 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:39 smithi170 ceph-mon[23272]: Updating smithi077:/etc/ceph/ceph.conf 2024-05-06T13:53:40.046 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:39 smithi170 ceph-mon[23272]: Updating smithi170:/etc/ceph/ceph.conf 2024-05-06T13:53:40.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:39 smithi077 ceph-mon[22428]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:53:40.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:39 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:40.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:39 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:40.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:39 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:40.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:39 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:40.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:39 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:53:40.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:39 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:53:40.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:39 smithi077 ceph-mon[22428]: Updating smithi077:/etc/ceph/ceph.conf 2024-05-06T13:53:40.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:39 smithi077 ceph-mon[22428]: Updating smithi170:/etc/ceph/ceph.conf 2024-05-06T13:53:40.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:39 smithi077 ceph-mon[27166]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:53:40.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:39 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:40.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:39 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:40.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:39 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:40.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:39 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:40.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:39 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:53:40.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:39 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:53:40.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:39 smithi077 ceph-mon[27166]: Updating smithi077:/etc/ceph/ceph.conf 2024-05-06T13:53:40.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:39 smithi077 ceph-mon[27166]: Updating smithi170:/etc/ceph/ceph.conf 2024-05-06T13:53:40.321 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:40 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:53:40] ENGINE Bus STARTING 2024-05-06T13:53:40.321 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:40 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: CherryPy Checker: 2024-05-06T13:53:40.321 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:40 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: The Application mounted at '' has an empty config. 2024-05-06T13:53:40.321 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:40 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: 2024-05-06T13:53:40.321 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:40 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:53:40] ENGINE Serving on http://:::9283 2024-05-06T13:53:40.321 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:40 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[43052]: [06/May/2024:13:53:40] ENGINE Bus STARTED 2024-05-06T13:53:41.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[22428]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:53:41.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[22428]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:53:41.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[22428]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:53:41.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[22428]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:53:41.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[22428]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:53:41.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[22428]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:53:41.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:41.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:41.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:41.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:41.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:41.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[22428]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:53:41.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:53:41.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:53:41.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[22428]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:53:41.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[22428]: Standby manager daemon x started 2024-05-06T13:53:41.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.170:0/2351606795' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T13:53:41.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.170:0/2351606795' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:53:41.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.170:0/2351606795' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T13:53:41.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.170:0/2351606795' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:53:41.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[27166]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:53:41.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[27166]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:53:41.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[27166]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:53:41.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[27166]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:53:41.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[27166]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:53:41.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[27166]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:53:41.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:41.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:41.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:41.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:41.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:41.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[27166]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:53:41.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:53:41.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:53:41.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[27166]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:53:41.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[27166]: Standby manager daemon x started 2024-05-06T13:53:41.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/2351606795' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T13:53:41.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/2351606795' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:53:41.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/2351606795' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T13:53:41.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:40 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/2351606795' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:53:41.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:40 smithi170 ceph-mon[23272]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:53:41.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:40 smithi170 ceph-mon[23272]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:53:41.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:40 smithi170 ceph-mon[23272]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:53:41.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:40 smithi170 ceph-mon[23272]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:53:41.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:40 smithi170 ceph-mon[23272]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:53:41.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:40 smithi170 ceph-mon[23272]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:53:41.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:40 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:41.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:40 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:41.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:40 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:41.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:40 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:41.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:40 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:41.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:40 smithi170 ceph-mon[23272]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:53:41.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:40 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:53:41.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:40 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:53:41.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:40 smithi170 ceph-mon[23272]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:53:41.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:40 smithi170 ceph-mon[23272]: Standby manager daemon x started 2024-05-06T13:53:41.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:40 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.170:0/2351606795' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T13:53:41.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:40 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.170:0/2351606795' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:53:41.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:40 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.170:0/2351606795' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T13:53:41.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:40 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.170:0/2351606795' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:53:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:41 smithi170 ceph-mon[23272]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 0 B/s wr, 119 op/s 2024-05-06T13:53:42.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:41 smithi170 ceph-mon[23272]: mgrmap e41: y(active, since 9s), standbys: x 2024-05-06T13:53:42.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:41 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T13:53:42.145 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:41 smithi077 ceph-mon[22428]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 0 B/s wr, 119 op/s 2024-05-06T13:53:42.145 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:41 smithi077 ceph-mon[22428]: mgrmap e41: y(active, since 9s), standbys: x 2024-05-06T13:53:42.145 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:41 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T13:53:42.145 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:41 smithi077 ceph-mon[27166]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 0 B/s wr, 119 op/s 2024-05-06T13:53:42.145 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:41 smithi077 ceph-mon[27166]: mgrmap e41: y(active, since 9s), standbys: x 2024-05-06T13:53:42.146 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:41 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T13:53:43.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:42 smithi077 ceph-mon[22428]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:53:43.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:42 smithi077 ceph-mon[27166]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:53:43.303 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:53:42.966+0000 7fb08fc52700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:53:43.303 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Creating ceph-iscsi config... 2024-05-06T13:53:43.303 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:53:43.303 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:53:43.304 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:53:43.304 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:53:43.304 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:53:43.304 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Traceback (most recent call last): 2024-05-06T13:53:43.304 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:53:43.304 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: return _run_code(code, main_globals, None, 2024-05-06T13:53:43.304 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:53:43.304 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: exec(code, run_globals) 2024-05-06T13:53:43.304 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:53:43.304 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:53:43.304 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:53:43.304 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:53:43.304 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:53:43.304 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:53:43.304 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:53:43.305 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:53:43.305 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:53:43.305 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Traceback (most recent call last): 2024-05-06T13:53:43.305 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:53:43.305 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:53:43.305 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:53:43.305 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:53:43.305 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:53:43.305 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:53:43.305 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:53:43.305 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: return future.result(timeout) 2024-05-06T13:53:43.305 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:53:43.305 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: return self.__get_result() 2024-05-06T13:53:43.305 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:53:43.306 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: raise self._exception 2024-05-06T13:53:43.306 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:53:43.306 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: use_current_daemon_image=reconfig, 2024-05-06T13:53:43.306 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:53:43.306 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:53:43.306 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:53:43.306 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Creating ceph-iscsi config... 2024-05-06T13:53:43.306 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:53:43.306 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:53:43.306 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:53:43.306 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:53:43.306 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:53:43.306 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Traceback (most recent call last): 2024-05-06T13:53:43.306 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:53:43.306 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: return _run_code(code, main_globals, None, 2024-05-06T13:53:43.307 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:53:43.307 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: exec(code, run_globals) 2024-05-06T13:53:43.307 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:53:43.307 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:53:43.307 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:53:43.307 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:53:43.307 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:53:43.307 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:53:43.307 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:53:43.307 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:53:43.307 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:53:43.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:42 smithi170 ceph-mon[23272]: from='client.15459 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:53:44.033 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:43 smithi170 ceph-mon[23272]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 93 op/s 2024-05-06T13:53:44.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:43 smithi077 ceph-mon[22428]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 93 op/s 2024-05-06T13:53:44.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:43 smithi077 ceph-mon[27166]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 93 op/s 2024-05-06T13:53:45.146 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:44 smithi170 ceph-mon[23272]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:53:45.146 INFO:journalctl@ceph.mon.b.smithi170.stdout: Creating ceph-iscsi config... 2024-05-06T13:53:45.146 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:53:45.147 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:53:45.147 INFO:journalctl@ceph.mon.b.smithi170.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:53:45.147 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:53:45.147 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:53:45.147 INFO:journalctl@ceph.mon.b.smithi170.stdout: Traceback (most recent call last): 2024-05-06T13:53:45.147 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:53:45.147 INFO:journalctl@ceph.mon.b.smithi170.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:53:45.147 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:53:45.147 INFO:journalctl@ceph.mon.b.smithi170.stdout: exec(code, run_globals) 2024-05-06T13:53:45.147 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:53:45.147 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:53:45.147 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:53:45.148 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:53:45.148 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:53:45.148 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:53:45.148 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:53:45.148 INFO:journalctl@ceph.mon.b.smithi170.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:53:45.148 INFO:journalctl@ceph.mon.b.smithi170.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:53:45.148 INFO:journalctl@ceph.mon.b.smithi170.stdout: Traceback (most recent call last): 2024-05-06T13:53:45.148 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:53:45.148 INFO:journalctl@ceph.mon.b.smithi170.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:53:45.148 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:53:45.148 INFO:journalctl@ceph.mon.b.smithi170.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:53:45.148 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:53:45.148 INFO:journalctl@ceph.mon.b.smithi170.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:53:45.149 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:53:45.149 INFO:journalctl@ceph.mon.b.smithi170.stdout: return future.result(timeout) 2024-05-06T13:53:45.149 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:53:45.149 INFO:journalctl@ceph.mon.b.smithi170.stdout: return self.__get_result() 2024-05-06T13:53:45.149 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:53:45.149 INFO:journalctl@ceph.mon.b.smithi170.stdout: raise self._exception 2024-05-06T13:53:45.149 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:53:45.149 INFO:journalctl@ceph.mon.b.smithi170.stdout: use_current_daemon_image=reconfig, 2024-05-06T13:53:45.149 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:53:45.149 INFO:journalctl@ceph.mon.b.smithi170.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:53:45.149 INFO:journalctl@ceph.mon.b.smithi170.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:53:45.150 INFO:journalctl@ceph.mon.b.smithi170.stdout: Creating ceph-iscsi config... 2024-05-06T13:53:45.150 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:53:45.150 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:53:45.150 INFO:journalctl@ceph.mon.b.smithi170.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:53:45.150 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:53:45.150 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:53:45.150 INFO:journalctl@ceph.mon.b.smithi170.stdout: Traceback (most recent call last): 2024-05-06T13:53:45.150 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:53:45.150 INFO:journalctl@ceph.mon.b.smithi170.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:53:45.150 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:53:45.150 INFO:journalctl@ceph.mon.b.smithi170.stdout: exec(code, run_globals) 2024-05-06T13:53:45.150 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:53:45.150 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:53:45.150 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:53:45.150 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:53:45.151 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:53:45.151 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:53:45.151 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:53:45.151 INFO:journalctl@ceph.mon.b.smithi170.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:53:45.151 INFO:journalctl@ceph.mon.b.smithi170.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:53:45.151 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:44 smithi170 ceph-mon[23272]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-06T13:53:45.151 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:44 smithi170 ceph-mon[23272]: Reconfiguring daemon prometheus.a on smithi170 2024-05-06T13:53:45.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:44 smithi077 ceph-mon[22428]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:53:45.323 INFO:journalctl@ceph.mon.a.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:53:45.324 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:53:45.324 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:53:45.324 INFO:journalctl@ceph.mon.a.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:53:45.324 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:53:45.324 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:53:45.324 INFO:journalctl@ceph.mon.a.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:53:45.324 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:53:45.324 INFO:journalctl@ceph.mon.a.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:53:45.324 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:53:45.324 INFO:journalctl@ceph.mon.a.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:53:45.324 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:53:45.325 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:53:45.325 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:53:45.325 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:53:45.325 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:53:45.325 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:53:45.325 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:53:45.325 INFO:journalctl@ceph.mon.a.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:53:45.325 INFO:journalctl@ceph.mon.a.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:53:45.325 INFO:journalctl@ceph.mon.a.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:53:45.325 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:53:45.325 INFO:journalctl@ceph.mon.a.smithi077.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:53:45.325 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:53:45.325 INFO:journalctl@ceph.mon.a.smithi077.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:53:45.325 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:53:45.326 INFO:journalctl@ceph.mon.a.smithi077.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:53:45.326 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:53:45.326 INFO:journalctl@ceph.mon.a.smithi077.stdout: return future.result(timeout) 2024-05-06T13:53:45.326 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:53:45.326 INFO:journalctl@ceph.mon.a.smithi077.stdout: return self.__get_result() 2024-05-06T13:53:45.326 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:53:45.326 INFO:journalctl@ceph.mon.a.smithi077.stdout: raise self._exception 2024-05-06T13:53:45.326 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:53:45.326 INFO:journalctl@ceph.mon.a.smithi077.stdout: use_current_daemon_image=reconfig, 2024-05-06T13:53:45.326 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:53:45.326 INFO:journalctl@ceph.mon.a.smithi077.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:53:45.326 INFO:journalctl@ceph.mon.a.smithi077.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:53:45.326 INFO:journalctl@ceph.mon.a.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:53:45.326 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:53:45.327 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:53:45.327 INFO:journalctl@ceph.mon.a.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:53:45.327 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:53:45.327 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:53:45.327 INFO:journalctl@ceph.mon.a.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:53:45.327 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:53:45.327 INFO:journalctl@ceph.mon.a.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:53:45.327 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:53:45.327 INFO:journalctl@ceph.mon.a.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:53:45.327 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:53:45.327 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:53:45.327 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:53:45.327 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:53:45.327 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:53:45.327 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:53:45.328 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:53:45.328 INFO:journalctl@ceph.mon.a.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:53:45.328 INFO:journalctl@ceph.mon.a.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:53:45.328 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:44 smithi077 ceph-mon[22428]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-06T13:53:45.328 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:44 smithi077 ceph-mon[22428]: Reconfiguring daemon prometheus.a on smithi170 2024-05-06T13:53:45.328 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:44 smithi077 ceph-mon[27166]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:53:45.328 INFO:journalctl@ceph.mon.c.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:53:45.328 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:53:45.328 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:53:45.328 INFO:journalctl@ceph.mon.c.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:53:45.328 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:53:45.328 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:53:45.329 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:53:45.329 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:53:45.329 INFO:journalctl@ceph.mon.c.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:53:45.329 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:53:45.329 INFO:journalctl@ceph.mon.c.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:53:45.329 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:53:45.329 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:53:45.329 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:53:45.329 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:53:45.329 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:53:45.329 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:53:45.329 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:53:45.329 INFO:journalctl@ceph.mon.c.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:53:45.329 INFO:journalctl@ceph.mon.c.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:53:45.329 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:53:45.330 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:53:45.330 INFO:journalctl@ceph.mon.c.smithi077.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:53:45.330 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:53:45.330 INFO:journalctl@ceph.mon.c.smithi077.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:53:45.330 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:53:45.330 INFO:journalctl@ceph.mon.c.smithi077.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:53:45.330 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:53:45.330 INFO:journalctl@ceph.mon.c.smithi077.stdout: return future.result(timeout) 2024-05-06T13:53:45.330 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:53:45.330 INFO:journalctl@ceph.mon.c.smithi077.stdout: return self.__get_result() 2024-05-06T13:53:45.330 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:53:45.330 INFO:journalctl@ceph.mon.c.smithi077.stdout: raise self._exception 2024-05-06T13:53:45.330 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:53:45.330 INFO:journalctl@ceph.mon.c.smithi077.stdout: use_current_daemon_image=reconfig, 2024-05-06T13:53:45.331 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:53:45.331 INFO:journalctl@ceph.mon.c.smithi077.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:53:45.331 INFO:journalctl@ceph.mon.c.smithi077.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:53:45.331 INFO:journalctl@ceph.mon.c.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:53:45.331 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:53:45.331 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:53:45.331 INFO:journalctl@ceph.mon.c.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:53:45.331 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:53:45.331 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:53:45.331 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:53:45.331 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:53:45.331 INFO:journalctl@ceph.mon.c.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:53:45.331 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:53:45.331 INFO:journalctl@ceph.mon.c.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:53:45.331 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:53:45.332 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:53:45.332 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:53:45.332 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:53:45.332 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:53:45.332 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:53:45.332 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:53:45.332 INFO:journalctl@ceph.mon.c.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:53:45.332 INFO:journalctl@ceph.mon.c.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:53:45.332 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:44 smithi077 ceph-mon[27166]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-06T13:53:45.332 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:44 smithi077 ceph-mon[27166]: Reconfiguring daemon prometheus.a on smithi170 2024-05-06T13:53:46.303 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:45 smithi170 ceph-mon[23272]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s rd, 0 B/s wr, 141 op/s 2024-05-06T13:53:46.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:45 smithi077 ceph-mon[22428]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s rd, 0 B/s wr, 141 op/s 2024-05-06T13:53:46.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:45 smithi077 ceph-mon[27166]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s rd, 0 B/s wr, 141 op/s 2024-05-06T13:53:46.577 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:46 smithi170 systemd[1]: Stopping Ceph prometheus.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:53:46.577 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:46 smithi170 bash[58144]: Error: no container with name or ID "ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus.a" found: no such container 2024-05-06T13:53:46.999 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:46 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:53:46.582Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-05-06T13:53:47.000 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:46 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:53:46.582Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-05-06T13:53:47.000 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:46 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:53:46.582Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-05-06T13:53:47.000 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:46 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:53:46.582Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-05-06T13:53:47.000 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:46 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:53:46.582Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-05-06T13:53:47.000 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:46 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:53:46.582Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-05-06T13:53:47.000 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:46 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:53:46.582Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-05-06T13:53:47.000 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:46 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:53:46.582Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-05-06T13:53:47.000 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:46 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:53:46.582Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-05-06T13:53:47.000 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:46 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:53:46.582Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-05-06T13:53:47.000 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:46 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:53:46.582Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-05-06T13:53:47.000 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:46 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[55319]: ts=2024-05-06T13:53:46.582Z caller=main.go:1066 level=info msg="See you next time!" 2024-05-06T13:53:47.000 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:46 smithi170 podman[58157]: 2024-05-06 13:53:46.592669244 +0000 UTC m=+0.133611285 container died 7a5811bc6799e0a4b5031993e175a49d45053bdd299741b4699a7c112eff44f5 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:53:47.326 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:47 smithi170 podman[58157]: 2024-05-06 13:53:47.106796626 +0000 UTC m=+0.647738660 container cleanup 7a5811bc6799e0a4b5031993e175a49d45053bdd299741b4699a7c112eff44f5 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:53:47.326 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:47 smithi170 bash[58157]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a 2024-05-06T13:53:47.327 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:47 smithi170 bash[58185]: Error: no container with name or ID "ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus.a" found: no such container 2024-05-06T13:53:47.617 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:47 smithi170 podman[58171]: 2024-05-06 13:53:47.349611931 +0000 UTC m=+0.759074926 container remove 7a5811bc6799e0a4b5031993e175a49d45053bdd299741b4699a7c112eff44f5 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:53:47.617 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:47 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-05-06T13:53:47.617 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:47 smithi170 systemd[1]: Stopped Ceph prometheus.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:53:47.617 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:47 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@prometheus.a.service: Consumed 1.310s CPU time. 2024-05-06T13:53:47.617 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:47 smithi170 systemd[1]: Starting Ceph prometheus.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:53:47.617 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:47 smithi170 podman[58257]: 2024-05-06 13:53:47.494341716 +0000 UTC m=+0.019373951 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-05-06T13:53:47.617 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:47 smithi170 podman[58257]: 2024-05-06 13:53:47.617482288 +0000 UTC m=+0.142514518 container create 9cf6334e9077d456eed30726497fc97346030c4533c94714787e1c739fd0021f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:53:47.919 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:47 smithi170 ceph-mon[23272]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 129 op/s 2024-05-06T13:53:47.919 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:47 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:53:47.919 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:47 smithi170 podman[58257]: 2024-05-06 13:53:47.827747641 +0000 UTC m=+0.352779871 container init 9cf6334e9077d456eed30726497fc97346030c4533c94714787e1c739fd0021f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:53:47.919 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:47 smithi170 podman[58257]: 2024-05-06 13:53:47.830230883 +0000 UTC m=+0.355263114 container start 9cf6334e9077d456eed30726497fc97346030c4533c94714787e1c739fd0021f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:53:47.919 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:47 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:53:47.854Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-05-06T13:53:47.920 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:47 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:53:47.854Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-05-06T13:53:47.920 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:47 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:53:47.854Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-05-06T13:53:47.920 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:47 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:53:47.854Z caller=main.go:518 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi170 (none))" 2024-05-06T13:53:47.920 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:47 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:53:47.854Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-05-06T13:53:47.920 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:47 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:53:47.854Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-05-06T13:53:47.920 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:47 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:53:47.856Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-05-06T13:53:47.920 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:47 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:53:47.856Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-05-06T13:53:47.920 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:47 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:53:47.857Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-05-06T13:53:47.920 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:47 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:53:47.859Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-05-06T13:53:47.920 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:47 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:53:47.859Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.275µs 2024-05-06T13:53:47.920 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:47 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:53:47.859Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-05-06T13:53:47.920 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:47 smithi170 bash[58257]: 9cf6334e9077d456eed30726497fc97346030c4533c94714787e1c739fd0021f 2024-05-06T13:53:48.100 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:47 smithi077 ceph-mon[22428]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 129 op/s 2024-05-06T13:53:48.100 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:47 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:53:48.101 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:47 smithi077 ceph-mon[27166]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 129 op/s 2024-05-06T13:53:48.101 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:47 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:53:48.101 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:47 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:53:47] ENGINE Bus STOPPING 2024-05-06T13:53:48.101 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:47 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:53:47] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-06T13:53:48.101 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:47 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:53:47] ENGINE Bus STOPPED 2024-05-06T13:53:48.101 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:47 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:53:47] ENGINE Bus STARTING 2024-05-06T13:53:48.101 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:48 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:53:48] ENGINE Serving on http://:::9283 2024-05-06T13:53:48.101 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:48 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:53:48] ENGINE Bus STARTED 2024-05-06T13:53:48.279 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:47 smithi170 systemd[1]: Started Ceph prometheus.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:53:48.279 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:53:48.046Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=5 2024-05-06T13:53:48.577 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:53:48.279Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=5 2024-05-06T13:53:48.577 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:53:48.342Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=5 2024-05-06T13:53:48.577 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:53:48.390Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=5 2024-05-06T13:53:48.577 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:53:48.394Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=5 2024-05-06T13:53:48.577 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:53:48.394Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=5 2024-05-06T13:53:48.578 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:53:48.394Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=44.346µs wal_replay_duration=534.994044ms total_replay_duration=535.053612ms 2024-05-06T13:53:48.578 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:53:48.398Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-05-06T13:53:48.578 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:53:48.398Z caller=main.go:947 level=info msg="TSDB started" 2024-05-06T13:53:48.578 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:53:48.398Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-05-06T13:53:48.578 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:53:48.419Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=21.499156ms db_storage=1.275µs remote_storage=1.376µs web_handler=300ns query_engine=640ns scrape=1.071331ms scrape_sd=155.871µs notify=18.525µs notify_sd=17.34µs rules=19.925483ms 2024-05-06T13:53:48.578 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:53:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:53:48.419Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-05-06T13:53:49.251 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:48 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:49.252 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:48 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:49.252 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:48 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:53:49.252 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:48 smithi170 ceph-mon[23272]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:53:49.252 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:48 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi170.front.sepia.ceph.com:9095"}]: dispatch 2024-05-06T13:53:49.252 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:48 smithi170 ceph-mon[23272]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi170.front.sepia.ceph.com:9095"}]: dispatch 2024-05-06T13:53:49.252 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:48 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:49.252 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:48 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:53:49.252 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:48 smithi170 ceph-mon[23272]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 129 op/s 2024-05-06T13:53:49.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:48 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:49.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:48 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:49.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:48 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:53:49.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:48 smithi077 ceph-mon[22428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:53:49.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:48 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi170.front.sepia.ceph.com:9095"}]: dispatch 2024-05-06T13:53:49.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:48 smithi077 ceph-mon[22428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi170.front.sepia.ceph.com:9095"}]: dispatch 2024-05-06T13:53:49.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:48 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:49.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:48 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:53:49.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:48 smithi077 ceph-mon[22428]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 129 op/s 2024-05-06T13:53:49.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:48 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:49.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:48 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:49.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:48 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:53:49.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:48 smithi077 ceph-mon[27166]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:53:49.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:48 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi170.front.sepia.ceph.com:9095"}]: dispatch 2024-05-06T13:53:49.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:48 smithi077 ceph-mon[27166]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi170.front.sepia.ceph.com:9095"}]: dispatch 2024-05-06T13:53:49.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:48 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:49.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:48 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:53:49.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:48 smithi077 ceph-mon[27166]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 129 op/s 2024-05-06T13:53:50.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:49 smithi077 ceph-mon[22428]: Upgrade: Updating mgr.x 2024-05-06T13:53:50.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:49 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T13:53:50.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:49 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T13:53:50.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:49 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:53:50.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:49 smithi077 ceph-mon[22428]: Deploying daemon mgr.x on smithi170 2024-05-06T13:53:50.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:49 smithi077 ceph-mon[27166]: Upgrade: Updating mgr.x 2024-05-06T13:53:50.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:49 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T13:53:50.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:49 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T13:53:50.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:49 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:53:50.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:49 smithi077 ceph-mon[27166]: Deploying daemon mgr.x on smithi170 2024-05-06T13:53:50.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:49 smithi170 ceph-mon[23272]: Upgrade: Updating mgr.x 2024-05-06T13:53:50.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:49 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T13:53:50.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:49 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T13:53:50.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:49 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:53:50.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:49 smithi170 ceph-mon[23272]: Deploying daemon mgr.x on smithi170 2024-05-06T13:53:51.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:50 smithi077 ceph-mon[22428]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 129 op/s 2024-05-06T13:53:51.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:50 smithi077 ceph-mon[27166]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 129 op/s 2024-05-06T13:53:51.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:50 smithi170 ceph-mon[23272]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 129 op/s 2024-05-06T13:53:51.661 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:51 smithi170 systemd[1]: Stopping Ceph mgr.x for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:53:51.922 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:51 smithi170 podman[58603]: 2024-05-06 13:53:51.660520928 +0000 UTC m=+0.135124246 container died 63ef6825ff68ea8352f7e7a157677daef5c84d1144b93dcc93ada99c06d81bae (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-70ce953, GIT_BRANCH=HEAD) 2024-05-06T13:53:52.181 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:51 smithi170 podman[58603]: 2024-05-06 13:53:51.987940343 +0000 UTC m=+0.462543660 container cleanup 63ef6825ff68ea8352f7e7a157677daef5c84d1144b93dcc93ada99c06d81bae (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-70ce953, GIT_BRANCH=HEAD) 2024-05-06T13:53:52.181 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:51 smithi170 bash[58603]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x 2024-05-06T13:53:52.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:52 smithi170 podman[58617]: 2024-05-06 13:53:52.180821801 +0000 UTC m=+0.523080857 container remove 63ef6825ff68ea8352f7e7a157677daef5c84d1144b93dcc93ada99c06d81bae (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-06T13:53:52.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:52 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-05-06T13:53:52.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:52 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-05-06T13:53:52.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:52 smithi170 systemd[1]: Stopped Ceph mgr.x for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:53:52.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:52 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mgr.x.service: Consumed 58.772s CPU time. 2024-05-06T13:53:52.947 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:52 smithi170 systemd[1]: Starting Ceph mgr.x for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:53:52.947 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:52 smithi170 podman[58748]: 2024-05-06 13:53:52.718624253 +0000 UTC m=+0.017818926 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:53:52.947 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:52 smithi170 podman[58748]: 2024-05-06 13:53:52.840607117 +0000 UTC m=+0.139801779 container create 125521be53a3c22765c4036f2d53d0ccb5fb2b05da7976421dbaf0b94c474aec (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-70ce953, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-05-06T13:53:53.426 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:53 smithi170 podman[58748]: 2024-05-06 13:53:53.076839687 +0000 UTC m=+0.376034347 container init 125521be53a3c22765c4036f2d53d0ccb5fb2b05da7976421dbaf0b94c474aec (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-06T13:53:53.426 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:53 smithi170 podman[58748]: 2024-05-06 13:53:53.07972684 +0000 UTC m=+0.378921503 container start 125521be53a3c22765c4036f2d53d0ccb5fb2b05da7976421dbaf0b94c474aec (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, ceph=True) 2024-05-06T13:53:53.426 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:53 smithi170 ceph-mgr[58766]: -- 172.21.15.170:0/1807668599 <== mon.2 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55850ce145a0 con 0x55850cde8c00 2024-05-06T13:53:53.426 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:53 smithi170 bash[58748]: 125521be53a3c22765c4036f2d53d0ccb5fb2b05da7976421dbaf0b94c474aec 2024-05-06T13:53:53.426 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:53 smithi170 systemd[1]: Started Ceph mgr.x for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:53:53.426 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:53 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:53:53.229+0000 7f056822b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T13:53:53.426 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:53 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:53:53.425+0000 7f056822b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:53:53.745 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:53 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:53:53.535+0000 7f056822b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T13:53:53.872 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:53 smithi077 ceph-mon[22428]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-05-06T13:53:53.872 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:53 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:53.872 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:53 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:53.873 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:53 smithi077 ceph-mon[27166]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-05-06T13:53:53.873 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:53 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:53.873 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:53 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:54.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:53 smithi170 ceph-mon[23272]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-05-06T13:53:54.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:53 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:54.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:53 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:54.077 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:53 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:53:53.882+0000 7f056822b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T13:53:54.077 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:53 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:53:53.972+0000 7f056822b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T13:53:55.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:54 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/4146207985' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:53:55.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:54 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/4146207985' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:53:55.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:54 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/4146207985' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:53:55.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:53:55.215+0000 7f056822b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T13:53:55.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:53:55.406+0000 7f056822b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T13:53:55.853 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:55 smithi170 ceph-mon[23272]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-05-06T13:53:55.853 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:53:55.593+0000 7f056822b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T13:53:56.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:55 smithi077 ceph-mon[22428]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-05-06T13:53:56.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:55 smithi077 ceph-mon[27166]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 59 op/s 2024-05-06T13:53:56.202 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:53:55.894+0000 7f056822b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T13:53:56.202 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:55 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:53:55.989+0000 7f056822b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T13:53:56.202 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:56 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:53:56.201+0000 7f056822b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T13:53:56.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:56 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:53:56.406+0000 7f056822b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:53:57.331 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:53:57.041+0000 7f056822b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T13:53:57.591 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:53:57.491+0000 7f056822b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T13:53:58.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:57 smithi077 ceph-mon[22428]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:53:58.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:57 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:58.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:57 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:58.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:57 smithi077 ceph-mon[27166]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:53:58.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:57 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:58.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:57 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:58.056 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:57 smithi170 ceph-mon[23272]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:53:58.057 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:57 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:58.057 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:57 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:53:58.057 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:53:57.590+0000 7f056822b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T13:53:58.057 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:57 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:53:57.808+0000 7f056822b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-06T13:53:58.736 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:58 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:53:58.372+0000 7f056822b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T13:53:58.736 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:58 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:53:58.473+0000 7f056822b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T13:53:58.991 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:58 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:53:58.735+0000 7f056822b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T13:53:58.992 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:58 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:53:58.883+0000 7f056822b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T13:53:59.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:58 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:53:58.991+0000 7f056822b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T13:53:59.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:59 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:53:59.117+0000 7f056822b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T13:53:59.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:53:59 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:53:59.216+0000 7f056822b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T13:53:59.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:53:59 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:53:59] "GET /metrics HTTP/1.1" 200 37988 "" "Prometheus/2.33.4" 2024-05-06T13:54:00.070 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:59 smithi170 ceph-mon[23272]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:54:00.070 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:59 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:00.070 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:53:59 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:00.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:59 smithi077 ceph-mon[27166]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:54:00.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:59 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:00.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:53:59 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:00.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:59 smithi077 ceph-mon[22428]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:54:00.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:59 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:00.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:53:59 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:00.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:54:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:54:00.069+0000 7f056822b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T13:54:00.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:54:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:54:00.295+0000 7f056822b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T13:54:00.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:54:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: [06/May/2024:13:54:00] ENGINE Bus STARTING 2024-05-06T13:54:00.708 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:54:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: CherryPy Checker: 2024-05-06T13:54:00.708 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:54:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: The Application mounted at '' has an empty config. 2024-05-06T13:54:00.708 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:54:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:54:00.708 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:54:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: [06/May/2024:13:54:00] ENGINE Serving on http://:::9283 2024-05-06T13:54:00.708 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:54:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: [06/May/2024:13:54:00] ENGINE Bus STARTED 2024-05-06T13:54:01.007 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:00 smithi170 ceph-mon[23272]: Standby manager daemon x restarted 2024-05-06T13:54:01.007 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:00 smithi170 ceph-mon[23272]: Standby manager daemon x started 2024-05-06T13:54:01.007 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:00 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.170:0/2987606547' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T13:54:01.007 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:00 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.170:0/2987606547' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:54:01.007 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:00 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.170:0/2987606547' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T13:54:01.007 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:00 smithi170 ceph-mon[23272]: from='mgr.? 172.21.15.170:0/2987606547' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:54:01.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:00 smithi077 ceph-mon[22428]: Standby manager daemon x restarted 2024-05-06T13:54:01.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:00 smithi077 ceph-mon[22428]: Standby manager daemon x started 2024-05-06T13:54:01.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:00 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.170:0/2987606547' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T13:54:01.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:00 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.170:0/2987606547' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:54:01.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:00 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.170:0/2987606547' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T13:54:01.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:00 smithi077 ceph-mon[22428]: from='mgr.? 172.21.15.170:0/2987606547' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:54:01.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:00 smithi077 ceph-mon[27166]: Standby manager daemon x restarted 2024-05-06T13:54:01.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:00 smithi077 ceph-mon[27166]: Standby manager daemon x started 2024-05-06T13:54:01.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:00 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/2987606547' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T13:54:01.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:00 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/2987606547' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:54:01.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:00 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/2987606547' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T13:54:01.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:00 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/2987606547' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:54:02.019 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:01 smithi170 ceph-mon[23272]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-05-06T13:54:02.020 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:01 smithi170 ceph-mon[23272]: mgrmap e42: y(active, since 29s), standbys: x 2024-05-06T13:54:02.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:01 smithi077 ceph-mon[22428]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-05-06T13:54:02.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:01 smithi077 ceph-mon[22428]: mgrmap e42: y(active, since 29s), standbys: x 2024-05-06T13:54:02.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:01 smithi077 ceph-mon[27166]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-05-06T13:54:02.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:01 smithi077 ceph-mon[27166]: mgrmap e42: y(active, since 29s), standbys: x 2024-05-06T13:54:03.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:03 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:03.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:03 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:03.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:03 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:54:03.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:03 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:54:03.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:03 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:03.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:03 smithi077 ceph-mon[22428]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:54:03.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:03 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:54:03.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:03 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:54:03.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:03 smithi077 ceph-mon[22428]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:54:03.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:03 smithi077 ceph-mon[22428]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-05-06T13:54:03.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:03 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:54:03.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:03 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:03.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:03 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:03.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:03 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:54:03.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:03 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:54:03.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:03 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:03.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:03 smithi077 ceph-mon[27166]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:54:03.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:03 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:54:03.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:03 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:54:03.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:03 smithi077 ceph-mon[27166]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:54:03.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:03 smithi077 ceph-mon[27166]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-05-06T13:54:03.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:03 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:54:03.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:03 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:03.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:03 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:03.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:03 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:54:03.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:03 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:54:03.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:03 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:03.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:03 smithi170 ceph-mon[23272]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:54:03.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:03 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:54:03.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:03 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:54:03.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:03 smithi170 ceph-mon[23272]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:54:03.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:03 smithi170 ceph-mon[23272]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-05-06T13:54:03.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:03 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:54:04.057 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T13:54:04.728 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T13:54:04.728 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (13m) 27s ago 21m 22.9M - 0.23.0 ba2b418f427c 0f855548777f 2024-05-06T13:54:04.729 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (13m) 7s ago 20m 53.4M - 8.3.5 dad864ee21e9 fdcecac79925 2024-05-06T13:54:04.729 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (3m) 27s ago 20m 50.9M - 3.5 e1d6a67b021e 79b44fb83b69 2024-05-06T13:54:04.729 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283,8765 running (12s) 7s ago 25m 68.7M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 125521be53a3 2024-05-06T13:54:04.729 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (9m) 27s ago 27m 475M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T13:54:04.729 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (27m) 27s ago 27m 72.1M 2048M 17.2.0 e1d6a67b021e ac46f91694f8 2024-05-06T13:54:04.729 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (25m) 7s ago 25m 55.1M 2048M 17.2.0 e1d6a67b021e 24be6fce9d28 2024-05-06T13:54:04.729 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (26m) 27s ago 26m 53.6M 2048M 17.2.0 e1d6a67b021e 83af336ef762 2024-05-06T13:54:04.729 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (13m) 27s ago 21m 20.5M - 1.3.1 1dbe0e931976 54aee5b22b1c 2024-05-06T13:54:04.729 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (13m) 7s ago 21m 18.0M - 1.3.1 1dbe0e931976 ca5df43a7469 2024-05-06T13:54:04.729 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (25m) 27s ago 25m 53.4M 2992M 17.2.0 e1d6a67b021e 2ccee2bb6e96 2024-05-06T13:54:04.729 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (25m) 27s ago 25m 57.3M 2992M 17.2.0 e1d6a67b021e 1d9e6609b585 2024-05-06T13:54:04.729 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (24m) 27s ago 24m 52.2M 2992M 17.2.0 e1d6a67b021e 37a61d295111 2024-05-06T13:54:04.729 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (24m) 27s ago 24m 51.5M 2992M 17.2.0 e1d6a67b021e 8feb7c077aaa 2024-05-06T13:54:04.730 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (23m) 7s ago 23m 57.9M 3248M 17.2.0 e1d6a67b021e f865acc44a06 2024-05-06T13:54:04.730 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (23m) 7s ago 23m 52.3M 3248M 17.2.0 e1d6a67b021e a55bd4122f6f 2024-05-06T13:54:04.730 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (23m) 7s ago 23m 51.1M 3248M 17.2.0 e1d6a67b021e 362938a0b357 2024-05-06T13:54:04.730 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (22m) 7s ago 22m 53.5M 3248M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T13:54:04.730 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (17s) 7s ago 22m 48.9M - 2.33.4 514e6a882f6e 9cf6334e9077 2024-05-06T13:54:04.730 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (20m) 27s ago 20m 76.9M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T13:54:04.730 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (20m) 7s ago 20m 75.6M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T13:54:05.142 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:54:05.142 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-05-06T13:54:05.143 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-05-06T13:54:05.143 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:54:05.143 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-05-06T13:54:05.143 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T13:54:05.143 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:54:05.143 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-05-06T13:54:05.143 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-06T13:54:05.143 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:54:05.143 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": {}, 2024-05-06T13:54:05.143 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw": { 2024-05-06T13:54:05.143 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T13:54:05.143 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:54:05.143 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-05-06T13:54:05.143 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-05-06T13:54:05.143 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T13:54:05.144 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T13:54:05.144 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:54:05.490 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:54:05.490 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T13:54:05.490 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-05-06T13:54:05.490 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-05-06T13:54:05.490 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [ 2024-05-06T13:54:05.490 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr" 2024-05-06T13:54:05.490 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-05-06T13:54:05.490 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "2/2 daemons upgraded", 2024-05-06T13:54:05.490 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "Currently upgrading mgr daemons", 2024-05-06T13:54:05.490 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-05-06T13:54:05.491 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:54:06.053 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:54:05.667+0000 7fb08fc52700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:54:06.053 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Creating ceph-iscsi config... 2024-05-06T13:54:06.053 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:54:06.053 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:54:06.053 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:54:06.053 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:54:06.053 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:54:06.053 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Traceback (most recent call last): 2024-05-06T13:54:06.053 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:54:06.053 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: return _run_code(code, main_globals, None, 2024-05-06T13:54:06.053 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:54:06.053 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: exec(code, run_globals) 2024-05-06T13:54:06.054 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:54:06.054 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:54:06.054 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:54:06.054 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:54:06.054 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:54:06.054 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:54:06.054 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:54:06.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[22428]: from='client.25519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:54:06.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[22428]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:54:06.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[22428]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:54:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[22428]: from='client.25531 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:54:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/893780732' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:54:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-05-06T13:54:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-05-06T13:54:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-05-06T13:54:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-05-06T13:54:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:06.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:06.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:06.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:06.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:06.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:06.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:06.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:06.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:06.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:06.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[27166]: from='client.25519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:54:06.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[27166]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:54:06.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[27166]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:54:06.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[27166]: from='client.25531 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:54:06.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/893780732' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:06.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:54:06.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:06.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:06.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-05-06T13:54:06.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-05-06T13:54:06.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-05-06T13:54:06.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-05-06T13:54:06.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:06.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:06.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:06.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:06.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:06.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:06.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:06.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:06.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:06.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:06.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:06.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:06.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:06.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:05 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:06.059 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:54:06.059 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:54:06.059 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Traceback (most recent call last): 2024-05-06T13:54:06.059 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:54:06.059 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:54:06.059 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:54:06.059 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:54:06.059 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:54:06.059 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:54:06.059 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:54:06.059 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: return future.result(timeout) 2024-05-06T13:54:06.060 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:54:06.060 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: return self.__get_result() 2024-05-06T13:54:06.060 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:54:06.060 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: raise self._exception 2024-05-06T13:54:06.060 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:54:06.060 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: use_current_daemon_image=reconfig, 2024-05-06T13:54:06.060 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:54:06.060 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:54:06.060 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:54:06.060 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Creating ceph-iscsi config... 2024-05-06T13:54:06.060 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:54:06.060 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:54:06.060 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:54:06.060 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:54:06.060 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:54:06.061 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: Traceback (most recent call last): 2024-05-06T13:54:06.061 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:54:06.061 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: return _run_code(code, main_globals, None, 2024-05-06T13:54:06.061 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:54:06.062 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: exec(code, run_globals) 2024-05-06T13:54:06.062 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:54:06.062 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:54:06.062 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:54:06.062 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:54:06.062 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:54:06.062 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:54:06.062 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:54:06.062 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:54:06.062 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:54:06.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:05 smithi170 ceph-mon[23272]: from='client.25519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:54:06.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:05 smithi170 ceph-mon[23272]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:54:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:05 smithi170 ceph-mon[23272]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:54:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:05 smithi170 ceph-mon[23272]: from='client.25531 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:54:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:05 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/893780732' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:05 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:54:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:05 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:05 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:05 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-05-06T13:54:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:05 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-05-06T13:54:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:05 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-05-06T13:54:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:05 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-05-06T13:54:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:05 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:05 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:05 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:06.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:05 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:06.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:05 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:06.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:05 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:06.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:05 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:06.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:05 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:06.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:05 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:06.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:05 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:06.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:05 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:06.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:05 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:06.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:05 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:06.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:05 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:06.947 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:06 smithi077 ceph-mon[22428]: from='client.25543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:54:06.947 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:06 smithi077 ceph-mon[22428]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:54:06.947 INFO:journalctl@ceph.mon.a.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:54:06.947 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:54:06.947 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:54:06.947 INFO:journalctl@ceph.mon.a.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:54:06.947 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:54:06.947 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:54:06.947 INFO:journalctl@ceph.mon.a.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:54:06.947 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:54:06.947 INFO:journalctl@ceph.mon.a.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:54:06.947 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:54:06.947 INFO:journalctl@ceph.mon.a.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:54:06.948 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:54:06.948 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:54:06.948 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:54:06.948 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:54:06.948 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:54:06.948 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:54:06.948 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:54:06.948 INFO:journalctl@ceph.mon.a.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:54:06.948 INFO:journalctl@ceph.mon.a.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:54:06.948 INFO:journalctl@ceph.mon.a.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:54:06.948 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:54:06.948 INFO:journalctl@ceph.mon.a.smithi077.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:54:06.948 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:54:06.948 INFO:journalctl@ceph.mon.a.smithi077.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:54:06.949 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:54:06.949 INFO:journalctl@ceph.mon.a.smithi077.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:54:06.949 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:54:06.949 INFO:journalctl@ceph.mon.a.smithi077.stdout: return future.result(timeout) 2024-05-06T13:54:06.949 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:54:06.949 INFO:journalctl@ceph.mon.a.smithi077.stdout: return self.__get_result() 2024-05-06T13:54:06.949 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:54:06.949 INFO:journalctl@ceph.mon.a.smithi077.stdout: raise self._exception 2024-05-06T13:54:06.949 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:54:06.949 INFO:journalctl@ceph.mon.a.smithi077.stdout: use_current_daemon_image=reconfig, 2024-05-06T13:54:06.949 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:54:06.949 INFO:journalctl@ceph.mon.a.smithi077.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:54:06.949 INFO:journalctl@ceph.mon.a.smithi077.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:54:06.949 INFO:journalctl@ceph.mon.a.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:54:06.949 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:54:06.949 INFO:journalctl@ceph.mon.a.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:54:06.950 INFO:journalctl@ceph.mon.a.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:54:06.950 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:54:06.950 INFO:journalctl@ceph.mon.a.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:54:06.950 INFO:journalctl@ceph.mon.a.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:54:06.950 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:54:06.950 INFO:journalctl@ceph.mon.a.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:54:06.950 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:54:06.950 INFO:journalctl@ceph.mon.a.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:54:06.950 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:54:06.950 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:54:06.950 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:54:06.950 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:54:06.951 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:06 smithi077 ceph-mon[27166]: from='client.25543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:54:06.951 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:06 smithi077 ceph-mon[27166]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:54:06.951 INFO:journalctl@ceph.mon.c.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:54:06.951 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:54:06.951 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:54:06.951 INFO:journalctl@ceph.mon.c.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:54:06.951 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:54:06.951 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:54:06.951 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:54:06.951 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:54:06.951 INFO:journalctl@ceph.mon.c.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:54:06.951 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:54:06.951 INFO:journalctl@ceph.mon.c.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:54:06.951 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:54:06.952 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:54:06.952 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:54:06.952 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:54:06.952 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:54:06.952 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:54:06.952 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:54:06.952 INFO:journalctl@ceph.mon.c.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:54:06.952 INFO:journalctl@ceph.mon.c.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:54:06.952 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:54:06.952 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:54:06.952 INFO:journalctl@ceph.mon.c.smithi077.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:54:06.952 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:54:06.952 INFO:journalctl@ceph.mon.c.smithi077.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:54:06.952 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:54:06.953 INFO:journalctl@ceph.mon.c.smithi077.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:54:06.953 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:54:06.953 INFO:journalctl@ceph.mon.c.smithi077.stdout: return future.result(timeout) 2024-05-06T13:54:06.953 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:54:06.953 INFO:journalctl@ceph.mon.c.smithi077.stdout: return self.__get_result() 2024-05-06T13:54:06.953 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:54:06.953 INFO:journalctl@ceph.mon.c.smithi077.stdout: raise self._exception 2024-05-06T13:54:06.953 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:54:06.953 INFO:journalctl@ceph.mon.c.smithi077.stdout: use_current_daemon_image=reconfig, 2024-05-06T13:54:06.953 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:54:06.953 INFO:journalctl@ceph.mon.c.smithi077.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:54:06.953 INFO:journalctl@ceph.mon.c.smithi077.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:54:06.953 INFO:journalctl@ceph.mon.c.smithi077.stdout: Creating ceph-iscsi config... 2024-05-06T13:54:06.953 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:54:06.954 INFO:journalctl@ceph.mon.c.smithi077.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:54:06.954 INFO:journalctl@ceph.mon.c.smithi077.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:54:06.954 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:54:06.954 INFO:journalctl@ceph.mon.c.smithi077.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:54:06.954 INFO:journalctl@ceph.mon.c.smithi077.stdout: Traceback (most recent call last): 2024-05-06T13:54:06.954 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:54:06.954 INFO:journalctl@ceph.mon.c.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:54:06.954 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:54:06.954 INFO:journalctl@ceph.mon.c.smithi077.stdout: exec(code, run_globals) 2024-05-06T13:54:06.954 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:54:06.954 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:54:06.954 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:54:06.954 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:54:06.955 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:54:06.955 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:54:06.955 INFO:journalctl@ceph.mon.a.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:54:06.955 INFO:journalctl@ceph.mon.a.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:54:06.955 INFO:journalctl@ceph.mon.a.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:54:06.955 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:06 smithi077 ceph-mon[22428]: Upgrade: Setting container_image for all mgr 2024-05-06T13:54:06.955 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:06 smithi077 ceph-mon[22428]: Upgrade: Setting container_image for all crash 2024-05-06T13:54:06.955 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:06 smithi077 ceph-mon[22428]: Upgrade: Setting container_image for all mds 2024-05-06T13:54:06.955 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:06 smithi077 ceph-mon[22428]: Upgrade: Setting container_image for all rbd-mirror 2024-05-06T13:54:06.955 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:06 smithi077 ceph-mon[22428]: Upgrade: Setting container_image for all cephfs-mirror 2024-05-06T13:54:06.955 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:06 smithi077 ceph-mon[22428]: Upgrade: Setting container_image for all ceph-exporter 2024-05-06T13:54:06.956 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:06 smithi077 ceph-mon[22428]: Upgrade: Setting container_image for all nfs 2024-05-06T13:54:06.956 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:06 smithi077 ceph-mon[22428]: Upgrade: Setting container_image for all nvmeof 2024-05-06T13:54:06.956 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:54:06.956 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:54:06.956 INFO:journalctl@ceph.mon.c.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:54:06.956 INFO:journalctl@ceph.mon.c.smithi077.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:54:06.956 INFO:journalctl@ceph.mon.c.smithi077.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:54:06.956 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:06 smithi077 ceph-mon[27166]: Upgrade: Setting container_image for all mgr 2024-05-06T13:54:06.956 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:06 smithi077 ceph-mon[27166]: Upgrade: Setting container_image for all crash 2024-05-06T13:54:06.956 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:06 smithi077 ceph-mon[27166]: Upgrade: Setting container_image for all mds 2024-05-06T13:54:06.956 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:06 smithi077 ceph-mon[27166]: Upgrade: Setting container_image for all rbd-mirror 2024-05-06T13:54:06.956 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:06 smithi077 ceph-mon[27166]: Upgrade: Setting container_image for all cephfs-mirror 2024-05-06T13:54:06.957 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:06 smithi077 ceph-mon[27166]: Upgrade: Setting container_image for all ceph-exporter 2024-05-06T13:54:06.957 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:06 smithi077 ceph-mon[27166]: Upgrade: Setting container_image for all nfs 2024-05-06T13:54:06.957 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:06 smithi077 ceph-mon[27166]: Upgrade: Setting container_image for all nvmeof 2024-05-06T13:54:07.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:06 smithi170 ceph-mon[23272]: from='client.25543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:54:07.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:06 smithi170 ceph-mon[23272]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:54:07.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: Creating ceph-iscsi config... 2024-05-06T13:54:07.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:54:07.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:54:07.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:54:07.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:54:07.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:54:07.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: Traceback (most recent call last): 2024-05-06T13:54:07.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:54:07.077 INFO:journalctl@ceph.mon.b.smithi170.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:54:07.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:54:07.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: exec(code, run_globals) 2024-05-06T13:54:07.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:54:07.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:54:07.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:54:07.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:54:07.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:54:07.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:54:07.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:54:07.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:54:07.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:54:07.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: Traceback (most recent call last): 2024-05-06T13:54:07.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-05-06T13:54:07.078 INFO:journalctl@ceph.mon.b.smithi170.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-05-06T13:54:07.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2455, in _daemon_action 2024-05-06T13:54:07.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-05-06T13:54:07.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 777, in wait_async 2024-05-06T13:54:07.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: return self.event_loop.get_result(coro, timeout) 2024-05-06T13:54:07.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-05-06T13:54:07.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: return future.result(timeout) 2024-05-06T13:54:07.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-05-06T13:54:07.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: return self.__get_result() 2024-05-06T13:54:07.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-05-06T13:54:07.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: raise self._exception 2024-05-06T13:54:07.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-05-06T13:54:07.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: use_current_daemon_image=reconfig, 2024-05-06T13:54:07.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-05-06T13:54:07.079 INFO:journalctl@ceph.mon.b.smithi170.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-05-06T13:54:07.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T13:54:07.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: Creating ceph-iscsi config... 2024-05-06T13:54:07.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T13:54:07.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T13:54:07.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: Non-zero exit code 1 from systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T13:54:07.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:54:07.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:54:07.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: Traceback (most recent call last): 2024-05-06T13:54:07.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T13:54:07.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: return _run_code(code, main_globals, None, 2024-05-06T13:54:07.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T13:54:07.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: exec(code, run_globals) 2024-05-06T13:54:07.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T13:54:07.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T13:54:07.080 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T13:54:07.081 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T13:54:07.081 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T13:54:07.081 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1123, in deploy_daemon 2024-05-06T13:54:07.081 INFO:journalctl@ceph.mon.b.smithi170.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T13:54:07.081 INFO:journalctl@ceph.mon.b.smithi170.stdout: RuntimeError: Failed command: systemctl restart ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T13:54:07.081 INFO:journalctl@ceph.mon.b.smithi170.stdout: See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T13:54:07.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:06 smithi170 ceph-mon[23272]: Upgrade: Setting container_image for all mgr 2024-05-06T13:54:07.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:06 smithi170 ceph-mon[23272]: Upgrade: Setting container_image for all crash 2024-05-06T13:54:07.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:06 smithi170 ceph-mon[23272]: Upgrade: Setting container_image for all mds 2024-05-06T13:54:07.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:06 smithi170 ceph-mon[23272]: Upgrade: Setting container_image for all rbd-mirror 2024-05-06T13:54:07.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:06 smithi170 ceph-mon[23272]: Upgrade: Setting container_image for all cephfs-mirror 2024-05-06T13:54:07.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:06 smithi170 ceph-mon[23272]: Upgrade: Setting container_image for all ceph-exporter 2024-05-06T13:54:07.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:06 smithi170 ceph-mon[23272]: Upgrade: Setting container_image for all nfs 2024-05-06T13:54:07.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:06 smithi170 ceph-mon[23272]: Upgrade: Setting container_image for all nvmeof 2024-05-06T13:54:08.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:07 smithi077 ceph-mon[22428]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:54:08.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:07 smithi077 ceph-mon[27166]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:54:08.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:07 smithi170 ceph-mon[23272]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:54:08.552 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:08 smithi077 systemd[1]: Stopping Ceph node-exporter.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:54:08.553 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:08 smithi077 podman[76209]: 2024-05-06 13:54:08.328723354 +0000 UTC m=+0.106841730 container died 54aee5b22b1c8a81efb1e966694a1742bda9d691f9c9088d9f3dfc6b3907e93b (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T13:54:08.894 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:08 smithi077 ceph-mon[27166]: Upgrade: Updating node-exporter.a (1/2) 2024-05-06T13:54:08.895 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:08 smithi077 ceph-mon[27166]: Deploying daemon node-exporter.a on smithi077 2024-05-06T13:54:08.895 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:08 smithi077 ceph-mon[22428]: Upgrade: Updating node-exporter.a (1/2) 2024-05-06T13:54:08.895 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:08 smithi077 ceph-mon[22428]: Deploying daemon node-exporter.a on smithi077 2024-05-06T13:54:08.895 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:08 smithi077 podman[76209]: 2024-05-06 13:54:08.668295392 +0000 UTC m=+0.446413761 container cleanup 54aee5b22b1c8a81efb1e966694a1742bda9d691f9c9088d9f3dfc6b3907e93b (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T13:54:08.895 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:08 smithi077 bash[76209]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a 2024-05-06T13:54:09.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:08 smithi170 ceph-mon[23272]: Upgrade: Updating node-exporter.a (1/2) 2024-05-06T13:54:09.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:08 smithi170 ceph-mon[23272]: Deploying daemon node-exporter.a on smithi077 2024-05-06T13:54:09.226 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:08 smithi077 podman[76223]: 2024-05-06 13:54:08.894885756 +0000 UTC m=+0.566784576 container remove 54aee5b22b1c8a81efb1e966694a1742bda9d691f9c9088d9f3dfc6b3907e93b (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T13:54:09.226 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:08 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-05-06T13:54:09.226 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:09 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-05-06T13:54:09.226 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:09 smithi077 systemd[1]: Stopped Ceph node-exporter.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:54:09.227 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:09 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@node-exporter.a.service: Consumed 5.105s CPU time. 2024-05-06T13:54:09.495 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:09 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:54:09] "GET /metrics HTTP/1.1" 200 37989 "" "Prometheus/2.33.4" 2024-05-06T13:54:09.495 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:09 smithi077 systemd[1]: Starting Ceph node-exporter.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:54:09.792 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:09 smithi077 bash[76354]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-05-06T13:54:10.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:09 smithi077 ceph-mon[22428]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:54:10.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:09 smithi077 ceph-mon[27166]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:54:10.053 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:09 smithi077 bash[76354]: Getting image source signatures 2024-05-06T13:54:10.053 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:09 smithi077 bash[76354]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-05-06T13:54:10.053 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:09 smithi077 bash[76354]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-05-06T13:54:10.053 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:09 smithi077 bash[76354]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-05-06T13:54:10.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:09 smithi170 ceph-mon[23272]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:54:12.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:11 smithi077 ceph-mon[22428]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 1 op/s 2024-05-06T13:54:12.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:11 smithi077 ceph-mon[27166]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 1 op/s 2024-05-06T13:54:12.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:11 smithi170 ceph-mon[23272]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 1 op/s 2024-05-06T13:54:12.552 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 bash[76354]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-05-06T13:54:12.552 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 bash[76354]: Writing manifest to image destination 2024-05-06T13:54:12.949 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 podman[76354]: 2024-05-06 13:54:12.615881376 +0000 UTC m=+3.222578365 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-05-06T13:54:12.950 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 podman[76354]: 2024-05-06 13:54:12.714250147 +0000 UTC m=+3.320947130 container create 67f45fa4968b50f9937d0731b271a5cef5e060f0f9563e0ac5a542212f6adbcf (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T13:54:12.950 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 podman[76354]: 2024-05-06 13:54:12.949762353 +0000 UTC m=+3.556459337 container init 67f45fa4968b50f9937d0731b271a5cef5e060f0f9563e0ac5a542212f6adbcf (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T13:54:13.302 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 podman[76354]: 2024-05-06 13:54:12.952862542 +0000 UTC m=+3.559559531 container start 67f45fa4968b50f9937d0731b271a5cef5e060f0f9563e0ac5a542212f6adbcf (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T13:54:13.303 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.957Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-05-06T13:54:13.303 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.957Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-05-06T13:54:13.303 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.957Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-05-06T13:54:13.303 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.957Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-05-06T13:54:13.303 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.958Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-05-06T13:54:13.303 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.958Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-05-06T13:54:13.303 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.958Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-05-06T13:54:13.303 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.958Z caller=node_exporter.go:117 level=info collector=arp 2024-05-06T13:54:13.303 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.958Z caller=node_exporter.go:117 level=info collector=bcache 2024-05-06T13:54:13.303 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=bonding 2024-05-06T13:54:13.303 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=btrfs 2024-05-06T13:54:13.303 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=conntrack 2024-05-06T13:54:13.304 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=cpu 2024-05-06T13:54:13.304 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-05-06T13:54:13.304 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=diskstats 2024-05-06T13:54:13.304 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=dmi 2024-05-06T13:54:13.304 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=edac 2024-05-06T13:54:13.304 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=entropy 2024-05-06T13:54:13.304 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-05-06T13:54:13.304 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=filefd 2024-05-06T13:54:13.305 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=filesystem 2024-05-06T13:54:13.305 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=hwmon 2024-05-06T13:54:13.305 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=infiniband 2024-05-06T13:54:13.305 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=ipvs 2024-05-06T13:54:13.305 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=loadavg 2024-05-06T13:54:13.305 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=mdadm 2024-05-06T13:54:13.305 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=meminfo 2024-05-06T13:54:13.305 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=netclass 2024-05-06T13:54:13.305 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=netdev 2024-05-06T13:54:13.305 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=netstat 2024-05-06T13:54:13.305 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=nfs 2024-05-06T13:54:13.305 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=nfsd 2024-05-06T13:54:13.305 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=nvme 2024-05-06T13:54:13.305 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=os 2024-05-06T13:54:13.305 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-05-06T13:54:13.306 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=pressure 2024-05-06T13:54:13.306 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=rapl 2024-05-06T13:54:13.306 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=schedstat 2024-05-06T13:54:13.306 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=selinux 2024-05-06T13:54:13.306 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=sockstat 2024-05-06T13:54:13.306 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=softnet 2024-05-06T13:54:13.306 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=stat 2024-05-06T13:54:13.306 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=tapestats 2024-05-06T13:54:13.306 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=textfile 2024-05-06T13:54:13.306 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-05-06T13:54:13.306 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=time 2024-05-06T13:54:13.306 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-05-06T13:54:13.306 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=uname 2024-05-06T13:54:13.306 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=vmstat 2024-05-06T13:54:13.306 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=xfs 2024-05-06T13:54:13.307 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=node_exporter.go:117 level=info collector=zfs 2024-05-06T13:54:13.307 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-05-06T13:54:13.307 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:12 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a[76403]: ts=2024-05-06T13:54:12.959Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-05-06T13:54:13.307 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:13 smithi077 bash[76354]: 67f45fa4968b50f9937d0731b271a5cef5e060f0f9563e0ac5a542212f6adbcf 2024-05-06T13:54:13.307 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 13:54:13 smithi077 systemd[1]: Started Ceph node-exporter.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:54:14.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:13 smithi077 ceph-mon[22428]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-06T13:54:14.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:13 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:14.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:13 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:14.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:13 smithi077 ceph-mon[27166]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-06T13:54:14.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:13 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:14.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:13 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:14.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:13 smithi170 ceph-mon[23272]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-06T13:54:14.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:13 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:14.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:13 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:15.801 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:15 smithi170 systemd[1]: Stopping Ceph node-exporter.b for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:54:16.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:15 smithi077 ceph-mon[27166]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-06T13:54:16.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:15 smithi077 ceph-mon[27166]: Upgrade: Updating node-exporter.b (2/2) 2024-05-06T13:54:16.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:15 smithi077 ceph-mon[27166]: Deploying daemon node-exporter.b on smithi170 2024-05-06T13:54:16.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:15 smithi077 ceph-mon[22428]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-06T13:54:16.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:15 smithi077 ceph-mon[22428]: Upgrade: Updating node-exporter.b (2/2) 2024-05-06T13:54:16.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:15 smithi077 ceph-mon[22428]: Deploying daemon node-exporter.b on smithi170 2024-05-06T13:54:16.076 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:15 smithi170 podman[60162]: 2024-05-06 13:54:15.891720485 +0000 UTC m=+0.136765764 container died ca5df43a746919ea7a70320f840927261e67e18cc1f650ae5a508281b9abe044 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T13:54:16.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:15 smithi170 ceph-mon[23272]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-06T13:54:16.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:15 smithi170 ceph-mon[23272]: Upgrade: Updating node-exporter.b (2/2) 2024-05-06T13:54:16.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:15 smithi170 ceph-mon[23272]: Deploying daemon node-exporter.b on smithi170 2024-05-06T13:54:16.548 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:16 smithi170 podman[60162]: 2024-05-06 13:54:16.23913983 +0000 UTC m=+0.484185116 container cleanup ca5df43a746919ea7a70320f840927261e67e18cc1f650ae5a508281b9abe044 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T13:54:16.549 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:16 smithi170 bash[60162]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b 2024-05-06T13:54:16.826 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:16 smithi170 podman[60176]: 2024-05-06 13:54:16.54900728 +0000 UTC m=+0.657630709 container remove ca5df43a746919ea7a70320f840927261e67e18cc1f650ae5a508281b9abe044 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T13:54:16.827 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:16 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-05-06T13:54:16.827 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:16 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-05-06T13:54:16.827 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:16 smithi170 systemd[1]: Stopped Ceph node-exporter.b for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:54:16.827 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:16 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@node-exporter.b.service: Consumed 5.415s CPU time. 2024-05-06T13:54:17.326 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:16 smithi170 systemd[1]: Starting Ceph node-exporter.b for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:54:17.326 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:17 smithi170 bash[60312]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-05-06T13:54:18.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:17 smithi170 ceph-mon[23272]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-05-06T13:54:18.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:17 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2094058210' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:54:18.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:17 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:54:18.077 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:17 smithi170 bash[60312]: Getting image source signatures 2024-05-06T13:54:18.077 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:17 smithi170 bash[60312]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-05-06T13:54:18.077 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:17 smithi170 bash[60312]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-05-06T13:54:18.077 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:17 smithi170 bash[60312]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-05-06T13:54:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:17 smithi077 ceph-mon[22428]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-05-06T13:54:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:17 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2094058210' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:54:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:17 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:54:18.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:17 smithi077 ceph-mon[27166]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-05-06T13:54:18.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:17 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2094058210' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:54:18.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:17 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:54:19.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:19 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:54:19] "GET /metrics HTTP/1.1" 200 37988 "" "Prometheus/2.33.4" 2024-05-06T13:54:20.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:19 smithi170 ceph-mon[23272]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-05-06T13:54:20.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:19 smithi077 ceph-mon[22428]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-05-06T13:54:20.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:19 smithi077 ceph-mon[27166]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-05-06T13:54:20.576 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 bash[60312]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-05-06T13:54:20.576 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 bash[60312]: Writing manifest to image destination 2024-05-06T13:54:20.954 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 podman[60312]: 2024-05-06 13:54:20.586488419 +0000 UTC m=+3.565230263 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-05-06T13:54:20.954 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 podman[60312]: 2024-05-06 13:54:20.69367013 +0000 UTC m=+3.672411980 container create 02081b71a5dd107771aa79ad7d5f31bc55e971aac33ebf9027b7380d044bfd77 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T13:54:20.954 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 podman[60312]: 2024-05-06 13:54:20.95426158 +0000 UTC m=+3.933003430 container init 02081b71a5dd107771aa79ad7d5f31bc55e971aac33ebf9027b7380d044bfd77 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T13:54:21.327 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 podman[60312]: 2024-05-06 13:54:20.957352246 +0000 UTC m=+3.936094090 container start 02081b71a5dd107771aa79ad7d5f31bc55e971aac33ebf9027b7380d044bfd77 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T13:54:21.327 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.961Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-05-06T13:54:21.327 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.961Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-05-06T13:54:21.327 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.961Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-05-06T13:54:21.327 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.961Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-05-06T13:54:21.327 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-05-06T13:54:21.327 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-05-06T13:54:21.327 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-05-06T13:54:21.327 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=arp 2024-05-06T13:54:21.328 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=bcache 2024-05-06T13:54:21.328 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=bonding 2024-05-06T13:54:21.328 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=btrfs 2024-05-06T13:54:21.328 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=conntrack 2024-05-06T13:54:21.328 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=cpu 2024-05-06T13:54:21.328 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-05-06T13:54:21.328 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=diskstats 2024-05-06T13:54:21.328 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=dmi 2024-05-06T13:54:21.329 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=edac 2024-05-06T13:54:21.329 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=entropy 2024-05-06T13:54:21.329 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-05-06T13:54:21.329 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=filefd 2024-05-06T13:54:21.329 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=filesystem 2024-05-06T13:54:21.329 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=hwmon 2024-05-06T13:54:21.329 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=infiniband 2024-05-06T13:54:21.329 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=ipvs 2024-05-06T13:54:21.329 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=loadavg 2024-05-06T13:54:21.329 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=mdadm 2024-05-06T13:54:21.329 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=meminfo 2024-05-06T13:54:21.329 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=netclass 2024-05-06T13:54:21.329 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=netdev 2024-05-06T13:54:21.329 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=netstat 2024-05-06T13:54:21.330 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=nfs 2024-05-06T13:54:21.330 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=nfsd 2024-05-06T13:54:21.330 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=nvme 2024-05-06T13:54:21.330 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=os 2024-05-06T13:54:21.330 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-05-06T13:54:21.330 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=pressure 2024-05-06T13:54:21.330 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=rapl 2024-05-06T13:54:21.330 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=schedstat 2024-05-06T13:54:21.330 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=selinux 2024-05-06T13:54:21.330 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=sockstat 2024-05-06T13:54:21.330 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=softnet 2024-05-06T13:54:21.330 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=stat 2024-05-06T13:54:21.330 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=tapestats 2024-05-06T13:54:21.330 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=textfile 2024-05-06T13:54:21.330 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-05-06T13:54:21.331 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=time 2024-05-06T13:54:21.331 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-05-06T13:54:21.331 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=uname 2024-05-06T13:54:21.331 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=vmstat 2024-05-06T13:54:21.331 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=xfs 2024-05-06T13:54:21.331 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.962Z caller=node_exporter.go:117 level=info collector=zfs 2024-05-06T13:54:21.331 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.963Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-05-06T13:54:21.331 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b[60361]: ts=2024-05-06T13:54:20.963Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-05-06T13:54:21.331 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:21 smithi170 bash[60312]: 02081b71a5dd107771aa79ad7d5f31bc55e971aac33ebf9027b7380d044bfd77 2024-05-06T13:54:21.331 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 13:54:21 smithi170 systemd[1]: Started Ceph node-exporter.b for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:54:22.060 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:21 smithi170 ceph-mon[23272]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-05-06T13:54:22.060 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:21 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:22.060 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:21 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:22.143 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:21 smithi077 ceph-mon[22428]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-05-06T13:54:22.143 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:21 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:22.143 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:21 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:22.143 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:21 smithi077 ceph-mon[27166]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-05-06T13:54:22.143 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:21 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:22.143 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:21 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:24.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:23 smithi170 ceph-mon[23272]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-05-06T13:54:24.260 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:23 smithi077 ceph-mon[22428]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-05-06T13:54:24.260 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:23 smithi077 ceph-mon[27166]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-05-06T13:54:26.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:25 smithi170 ceph-mon[23272]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-05-06T13:54:26.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:25 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:26.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:25 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:26.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:25 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:26.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:25 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:26.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:25 smithi077 ceph-mon[22428]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-05-06T13:54:26.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:25 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:26.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:25 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:26.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:25 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:26.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:25 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:26.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:25 smithi077 ceph-mon[27166]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-05-06T13:54:26.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:25 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:26.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:25 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:26.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:25 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:26.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:25 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:27.710 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:27 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:27.711 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:27 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:27.711 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:27 smithi170 ceph-mon[23272]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-05-06T13:54:27.711 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:27 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:27.711 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:27 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:27.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:27 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:27.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:27 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:27.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:27 smithi077 ceph-mon[22428]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-05-06T13:54:27.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:27 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:27.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:27 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:27.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:27 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:27.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:27 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:27.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:27 smithi077 ceph-mon[27166]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-05-06T13:54:27.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:27 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:27.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:27 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:28.616 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:28 smithi170 ceph-mon[23272]: from='client.25558 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:54:28.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:28 smithi077 ceph-mon[22428]: from='client.25558 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:54:28.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:28 smithi077 ceph-mon[27166]: from='client.25558 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:54:29.423 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:29 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:54:29] "GET /metrics HTTP/1.1" 200 37988 "" "Prometheus/2.33.4" 2024-05-06T13:54:29.738 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:29 smithi170 ceph-mon[23272]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-05-06T13:54:29.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:29 smithi077 ceph-mon[22428]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-05-06T13:54:29.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:29 smithi077 ceph-mon[27166]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-05-06T13:54:31.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:30 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:31.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:30 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:31.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:30 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:31.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:30 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:31.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:30 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:54:31.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:30 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:54:31.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:30 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:31.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:30 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:54:31.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:30 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:54:31.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:30 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:31.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:30 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:31.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:30 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:31.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:30 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:31.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:30 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:54:31.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:30 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:54:31.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:30 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:31.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:30 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:54:31.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:30 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:54:31.330 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:30 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:31.330 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:30 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:31.330 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:30 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:31.330 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:30 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:31.330 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:30 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:54:31.330 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:30 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:54:31.331 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:30 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:31.331 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:30 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T13:54:31.331 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:30 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:54:32.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:31 smithi077 ceph-mon[22428]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:54:32.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:31 smithi077 ceph-mon[22428]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:54:32.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:31 smithi077 ceph-mon[22428]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-05-06T13:54:32.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:31 smithi077 ceph-mon[27166]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:54:32.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:31 smithi077 ceph-mon[27166]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:54:32.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:31 smithi077 ceph-mon[27166]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-05-06T13:54:32.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:31 smithi170 ceph-mon[23272]: Reconfiguring iscsi.foo.smithi077.muddwz (dependencies changed)... 2024-05-06T13:54:32.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:31 smithi170 ceph-mon[23272]: Reconfiguring daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T13:54:32.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:31 smithi170 ceph-mon[23272]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-05-06T13:54:33.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:32 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:54:33.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:32 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:54:33.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:32 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:54:34.120 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:33 smithi077 ceph-mon[22428]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:54:34.120 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:33 smithi077 ceph-mon[27166]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:54:34.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:33 smithi170 ceph-mon[23272]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:54:35.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:35.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:35.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:54:35.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[27166]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:54:35.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:54:35.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[27166]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:54:35.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:54:35.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:35.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:35.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:35.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:35.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:35.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:35.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:35.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:35.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:35.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[27166]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:54:35.805 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/569470' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:54:35.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:35.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:35.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:54:35.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[22428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:54:35.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:54:35.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[22428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:54:35.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:54:35.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:35.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:35.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:35.806 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:35.806 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:35.806 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:35.806 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:35.806 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:35.806 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:35.806 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[22428]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:54:35.806 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:35 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/569470' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:54:35.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:35 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:35.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:35 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:35.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:35 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:54:35.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:35 smithi170 ceph-mon[23272]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-06T13:54:35.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:35 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:54:35.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:35 smithi170 ceph-mon[23272]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-06T13:54:35.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:35 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:54:35.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:35 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:35.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:35 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:35.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:35 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:35.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:35 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:35.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:35 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:35.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:35 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:35.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:35 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:35.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:35 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:35.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:35 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:35.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:35 smithi170 ceph-mon[23272]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:54:35.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:35 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/569470' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T13:54:35.844 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T13:54:36.501 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T13:54:36.501 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (13m) 11s ago 21m 22.9M - 0.23.0 ba2b418f427c 0f855548777f 2024-05-06T13:54:36.502 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (13m) 10s ago 21m 54.0M - 8.3.5 dad864ee21e9 fdcecac79925 2024-05-06T13:54:36.502 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 starting - - - - 2024-05-06T13:54:36.502 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283,8765 running (43s) 10s ago 26m 451M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 125521be53a3 2024-05-06T13:54:36.502 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (10m) 11s ago 27m 499M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T13:54:36.502 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (27m) 11s ago 27m 73.9M 2048M 17.2.0 e1d6a67b021e ac46f91694f8 2024-05-06T13:54:36.502 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (26m) 10s ago 26m 55.4M 2048M 17.2.0 e1d6a67b021e 24be6fce9d28 2024-05-06T13:54:36.502 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (26m) 11s ago 26m 55.7M 2048M 17.2.0 e1d6a67b021e 83af336ef762 2024-05-06T13:54:36.502 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (23s) 11s ago 22m 3397k - 1.5.0 0da6a335fe13 67f45fa4968b 2024-05-06T13:54:36.502 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (15s) 10s ago 22m 3338k - 1.5.0 0da6a335fe13 02081b71a5dd 2024-05-06T13:54:36.502 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (25m) 11s ago 25m 54.4M 2992M 17.2.0 e1d6a67b021e 2ccee2bb6e96 2024-05-06T13:54:36.502 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (25m) 11s ago 25m 58.5M 2992M 17.2.0 e1d6a67b021e 1d9e6609b585 2024-05-06T13:54:36.502 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (25m) 11s ago 25m 52.4M 2992M 17.2.0 e1d6a67b021e 37a61d295111 2024-05-06T13:54:36.502 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (24m) 11s ago 24m 52.9M 2992M 17.2.0 e1d6a67b021e 8feb7c077aaa 2024-05-06T13:54:36.502 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (24m) 10s ago 24m 58.1M 3248M 17.2.0 e1d6a67b021e f865acc44a06 2024-05-06T13:54:36.503 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (24m) 10s ago 24m 52.5M 3248M 17.2.0 e1d6a67b021e a55bd4122f6f 2024-05-06T13:54:36.503 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (23m) 10s ago 23m 51.2M 3248M 17.2.0 e1d6a67b021e 362938a0b357 2024-05-06T13:54:36.503 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (23m) 10s ago 23m 53.7M 3248M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T13:54:36.503 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (48s) 10s ago 22m 58.3M - 2.33.4 514e6a882f6e 9cf6334e9077 2024-05-06T13:54:36.503 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (21m) 11s ago 21m 77.1M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T13:54:36.503 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (20m) 10s ago 20m 76.0M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T13:54:36.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:36 smithi077 ceph-mon[22428]: from='client.15696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:54:36.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:36 smithi077 ceph-mon[27166]: from='client.15696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:54:36.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:36 smithi170 ceph-mon[23272]: from='client.15696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:54:36.893 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:54:36.893 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-05-06T13:54:36.893 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-05-06T13:54:36.893 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:54:36.893 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-05-06T13:54:36.893 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T13:54:36.893 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:54:36.893 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-05-06T13:54:36.893 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-06T13:54:36.894 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:54:36.894 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": {}, 2024-05-06T13:54:36.894 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw": { 2024-05-06T13:54:36.894 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T13:54:36.894 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:54:36.894 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-05-06T13:54:36.894 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-05-06T13:54:36.894 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T13:54:36.894 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T13:54:36.894 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:54:37.223 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:54:37.223 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T13:54:37.223 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-05-06T13:54:37.223 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-05-06T13:54:37.223 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [ 2024-05-06T13:54:37.223 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr" 2024-05-06T13:54:37.223 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-05-06T13:54:37.223 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "2/2 daemons upgraded", 2024-05-06T13:54:37.223 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "Currently upgrading prometheus daemons", 2024-05-06T13:54:37.224 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-05-06T13:54:37.224 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:54:37.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:37 smithi077 ceph-mon[22428]: Upgrade: Updating prometheus.a 2024-05-06T13:54:37.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:37 smithi077 ceph-mon[22428]: from='client.15702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:54:37.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:37 smithi077 ceph-mon[22428]: Deploying daemon prometheus.a on smithi170 2024-05-06T13:54:37.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:37 smithi077 ceph-mon[22428]: from='client.15708 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:54:37.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:37 smithi077 ceph-mon[22428]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:54:37.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:37 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2430668361' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:37.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:37 smithi077 ceph-mon[27166]: Upgrade: Updating prometheus.a 2024-05-06T13:54:37.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:37 smithi077 ceph-mon[27166]: from='client.15702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:54:37.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:37 smithi077 ceph-mon[27166]: Deploying daemon prometheus.a on smithi170 2024-05-06T13:54:37.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:37 smithi077 ceph-mon[27166]: from='client.15708 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:54:37.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:37 smithi077 ceph-mon[27166]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:54:37.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:37 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2430668361' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:37 smithi170 ceph-mon[23272]: Upgrade: Updating prometheus.a 2024-05-06T13:54:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:37 smithi170 ceph-mon[23272]: from='client.15702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:54:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:37 smithi170 ceph-mon[23272]: Deploying daemon prometheus.a on smithi170 2024-05-06T13:54:37.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:37 smithi170 ceph-mon[23272]: from='client.15708 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:54:37.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:37 smithi170 ceph-mon[23272]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:54:37.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:37 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2430668361' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:54:38.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:38 smithi077 ceph-mon[22428]: from='client.25621 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:54:38.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:38 smithi077 ceph-mon[27166]: from='client.25621 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:54:38.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:38 smithi170 ceph-mon[23272]: from='client.25621 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:54:39.516 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:54:39 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:54:39] "GET /metrics HTTP/1.1" 200 37984 "" "Prometheus/2.33.4" 2024-05-06T13:54:39.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:39 smithi077 ceph-mon[22428]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:54:39.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:39 smithi077 ceph-mon[27166]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:54:39.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:39 smithi170 ceph-mon[23272]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:54:42.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:41 smithi077 ceph-mon[22428]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:54:42.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:41 smithi077 ceph-mon[27166]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:54:42.061 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:41 smithi170 ceph-mon[23272]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:54:44.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:43 smithi077 ceph-mon[22428]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:54:44.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:43 smithi077 ceph-mon[27166]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:54:44.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:43 smithi170 ceph-mon[23272]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:54:46.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:45 smithi077 ceph-mon[22428]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:54:46.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:45 smithi077 ceph-mon[22428]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:54:46.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:45 smithi077 ceph-mon[27166]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:54:46.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:45 smithi077 ceph-mon[27166]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:54:46.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:45 smithi170 ceph-mon[23272]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:54:46.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:45 smithi170 ceph-mon[23272]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:54:48.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:47 smithi077 ceph-mon[22428]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:54:48.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:47 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:54:48.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:47 smithi077 ceph-mon[27166]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:54:48.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:47 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:54:48.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:47 smithi170 ceph-mon[23272]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:54:48.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:47 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:54:48.713 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:48 smithi170 systemd[1]: Stopping Ceph prometheus.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:54:49.261 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:54:48.976Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-05-06T13:54:49.261 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:54:48.977Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-05-06T13:54:49.261 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:54:48.977Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-05-06T13:54:49.261 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:54:48.977Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-05-06T13:54:49.261 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:54:48.977Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-05-06T13:54:49.261 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:54:48.977Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-05-06T13:54:49.261 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:54:48.977Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-05-06T13:54:49.261 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:54:48.977Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-05-06T13:54:49.261 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:54:48.977Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-05-06T13:54:49.261 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:54:48.977Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-05-06T13:54:49.261 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:54:48.977Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-05-06T13:54:49.261 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:48 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[58274]: ts=2024-05-06T13:54:48.977Z caller=main.go:1066 level=info msg="See you next time!" 2024-05-06T13:54:49.261 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:48 smithi170 podman[61865]: 2024-05-06 13:54:48.98728759 +0000 UTC m=+0.131110123 container died 9cf6334e9077d456eed30726497fc97346030c4533c94714787e1c739fd0021f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:54:49.576 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:49 smithi170 podman[61865]: 2024-05-06 13:54:49.351134522 +0000 UTC m=+0.494957051 container cleanup 9cf6334e9077d456eed30726497fc97346030c4533c94714787e1c739fd0021f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:54:49.576 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:49 smithi170 bash[61865]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a 2024-05-06T13:54:49.960 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:49 smithi170 ceph-mon[23272]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:54:49.960 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:49 smithi170 podman[61879]: 2024-05-06 13:54:49.635947832 +0000 UTC m=+0.650571204 container remove 9cf6334e9077d456eed30726497fc97346030c4533c94714787e1c739fd0021f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:54:49.961 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:49 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-05-06T13:54:49.961 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:49 smithi170 systemd[1]: Stopped Ceph prometheus.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:54:49.961 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:49 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@prometheus.a.service: Consumed 1.268s CPU time. 2024-05-06T13:54:50.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:49 smithi077 ceph-mon[22428]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:54:50.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:49 smithi077 ceph-mon[27166]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:54:50.326 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:49 smithi170 systemd[1]: Starting Ceph prometheus.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:54:50.326 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 podman[62014]: 2024-05-06 13:54:50.077773542 +0000 UTC m=+0.018835597 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-05-06T13:54:50.327 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 podman[62014]: 2024-05-06 13:54:50.187067653 +0000 UTC m=+0.128129719 container create 10d5c513bb9b635f812a95f30f073ab5eeb2b6d805f04f8ab31fafd543a5d7c4 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:54:50.827 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 podman[62014]: 2024-05-06 13:54:50.61654687 +0000 UTC m=+0.557608898 container init 10d5c513bb9b635f812a95f30f073ab5eeb2b6d805f04f8ab31fafd543a5d7c4 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:54:50.827 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 podman[62014]: 2024-05-06 13:54:50.619213752 +0000 UTC m=+0.560275776 container start 10d5c513bb9b635f812a95f30f073ab5eeb2b6d805f04f8ab31fafd543a5d7c4 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T13:54:50.827 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T13:54:50.642Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-05-06T13:54:50.827 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T13:54:50.642Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-05-06T13:54:50.827 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T13:54:50.642Z caller=main.go:570 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi170 (none))" 2024-05-06T13:54:50.827 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T13:54:50.642Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-05-06T13:54:50.828 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T13:54:50.642Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-05-06T13:54:50.828 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T13:54:50.643Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-05-06T13:54:50.828 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T13:54:50.644Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-05-06T13:54:50.828 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T13:54:50.646Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-05-06T13:54:50.828 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T13:54:50.646Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-05-06T13:54:50.828 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T13:54:50.647Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-05-06T13:54:50.828 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T13:54:50.647Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.47µs 2024-05-06T13:54:50.828 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T13:54:50.647Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-05-06T13:54:50.828 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T13:54:50.654Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=6 2024-05-06T13:54:50.828 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T13:54:50.668Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=6 2024-05-06T13:54:50.828 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T13:54:50.675Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=6 2024-05-06T13:54:50.828 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T13:54:50.678Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=6 2024-05-06T13:54:50.828 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T13:54:50.680Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=6 2024-05-06T13:54:50.828 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T13:54:50.681Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=6 2024-05-06T13:54:50.829 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T13:54:50.682Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=6 maxSegment=6 2024-05-06T13:54:50.829 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T13:54:50.682Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=43.777µs wal_replay_duration=34.236025ms wbl_replay_duration=130ns total_replay_duration=34.302796ms 2024-05-06T13:54:50.829 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 bash[62014]: 10d5c513bb9b635f812a95f30f073ab5eeb2b6d805f04f8ab31fafd543a5d7c4 2024-05-06T13:54:50.829 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T13:54:50.685Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-05-06T13:54:50.829 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T13:54:50.685Z caller=main.go:1029 level=info msg="TSDB started" 2024-05-06T13:54:50.829 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T13:54:50.685Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-05-06T13:54:50.829 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 systemd[1]: Started Ceph prometheus.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:54:50.829 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T13:54:50.705Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.793382ms db_storage=755ns remote_storage=1.245µs web_handler=341ns query_engine=1.12µs scrape=1.217717ms scrape_sd=102.431µs notify=16.449µs notify_sd=16.902µs rules=19.186243ms tracing=6.092µs 2024-05-06T13:54:50.829 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T13:54:50.705Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-05-06T13:54:50.829 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 13:54:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T13:54:50.705Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-05-06T13:54:52.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:51 smithi077 ceph-mon[22428]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:54:52.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:52.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:52.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:51 smithi077 ceph-mon[27166]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:54:52.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:52.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:52.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:51 smithi170 ceph-mon[23272]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:54:52.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:51 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:52.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:51 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:54.038 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:53 smithi170 ceph-mon[23272]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:54:54.043 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:53 smithi077 ceph-mon[22428]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:54:54.044 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:53 smithi077 ceph-mon[27166]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:54:56.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:55 smithi077 ceph-mon[27166]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:54:56.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:55 smithi077 ceph-mon[27166]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:54:56.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:55 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:56.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:55 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:56.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:55 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:56.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:55 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:56.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:55 smithi077 ceph-mon[22428]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:54:56.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:55 smithi077 ceph-mon[22428]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:54:56.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:55 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:56.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:55 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:56.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:55 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:56.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:55 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:56.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:55 smithi170 ceph-mon[23272]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:54:56.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:55 smithi170 ceph-mon[23272]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:54:56.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:55 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:56.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:55 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:56.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:55 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:56.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:55 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:57.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:57 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:57.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:57 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:57.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:57 smithi170 ceph-mon[23272]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:54:57.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:57 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:57.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:57 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:57.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:57 smithi077 ceph-mon[22428]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:54:57.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:57 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:57.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:57 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:54:57.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:57 smithi077 ceph-mon[27166]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:00.000 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:54:59 smithi170 ceph-mon[23272]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:00.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:54:59 smithi077 ceph-mon[22428]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:00.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:54:59 smithi077 ceph-mon[27166]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:01.539 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:01.540 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:01.540 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:55:01.540 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:55:01.540 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:01.540 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:55:01.540 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[22428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:55:01.540 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:55:01.540 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.540 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.540 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.540 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.540 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.540 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.541 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.541 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.541 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.541 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.541 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[22428]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:01.541 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:01.541 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:01.541 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:55:01.541 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:55:01.541 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:01.541 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:55:01.541 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[27166]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:55:01.541 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:55:01.542 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.542 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.542 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.542 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.542 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.542 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.542 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.542 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.542 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.542 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.542 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:01 smithi077 ceph-mon[27166]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:01.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:01 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:01.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:01 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:01.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:01 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:55:01.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:01 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:55:01.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:01 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:01.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:01 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:55:01.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:01 smithi170 ceph-mon[23272]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-05-06T13:55:01.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:01 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:55:01.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:01 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:01 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:01 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:01 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:01 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:01 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:01 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:01 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:01 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:01 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:01.828 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:01 smithi170 ceph-mon[23272]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:04.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:03 smithi077 ceph-mon[22428]: Upgrade: Updating alertmanager.a 2024-05-06T13:55:04.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:03 smithi077 ceph-mon[22428]: Deploying daemon alertmanager.a on smithi077 2024-05-06T13:55:04.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:03 smithi077 ceph-mon[22428]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:04.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:03 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:55:04.053 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:55:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:55:03] "GET /metrics HTTP/1.1" 200 37986 "" "Prometheus/2.43.0" 2024-05-06T13:55:04.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:03 smithi077 ceph-mon[27166]: Upgrade: Updating alertmanager.a 2024-05-06T13:55:04.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:03 smithi077 ceph-mon[27166]: Deploying daemon alertmanager.a on smithi077 2024-05-06T13:55:04.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:03 smithi077 ceph-mon[27166]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:04.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:03 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:55:04.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:03 smithi170 ceph-mon[23272]: Upgrade: Updating alertmanager.a 2024-05-06T13:55:04.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:03 smithi170 ceph-mon[23272]: Deploying daemon alertmanager.a on smithi077 2024-05-06T13:55:04.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:03 smithi170 ceph-mon[23272]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:04.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:03 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:55:06.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:05 smithi077 ceph-mon[22428]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:55:06.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:05 smithi077 ceph-mon[22428]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:06.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:05 smithi077 ceph-mon[27166]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:55:06.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:05 smithi077 ceph-mon[27166]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:06.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:05 smithi170 ceph-mon[23272]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:55:06.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:05 smithi170 ceph-mon[23272]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:07.580 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T13:55:08.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:07 smithi077 ceph-mon[22428]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:08.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:07 smithi077 ceph-mon[27166]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:08.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:07 smithi170 ceph-mon[23272]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:08.242 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T13:55:08.242 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (14m) 13s ago 22m 22.9M - 0.23.0 ba2b418f427c 0f855548777f 2024-05-06T13:55:08.242 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (14m) 13s ago 22m 54.0M - 8.3.5 dad864ee21e9 fdcecac79925 2024-05-06T13:55:08.242 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (34s) 13s ago 21m 52.3M - 3.5 e1d6a67b021e 9353a303fb55 2024-05-06T13:55:08.242 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283,8765 running (75s) 13s ago 26m 451M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 125521be53a3 2024-05-06T13:55:08.242 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (10m) 13s ago 28m 503M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T13:55:08.242 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (28m) 13s ago 28m 75.4M 2048M 17.2.0 e1d6a67b021e ac46f91694f8 2024-05-06T13:55:08.242 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (27m) 13s ago 27m 55.5M 2048M 17.2.0 e1d6a67b021e 24be6fce9d28 2024-05-06T13:55:08.242 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (27m) 13s ago 27m 55.4M 2048M 17.2.0 e1d6a67b021e 83af336ef762 2024-05-06T13:55:08.243 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (55s) 13s ago 23m 7818k - 1.5.0 0da6a335fe13 67f45fa4968b 2024-05-06T13:55:08.243 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (47s) 13s ago 22m 7314k - 1.5.0 0da6a335fe13 02081b71a5dd 2024-05-06T13:55:08.243 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (26m) 13s ago 26m 54.4M 2992M 17.2.0 e1d6a67b021e 2ccee2bb6e96 2024-05-06T13:55:08.243 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (26m) 13s ago 26m 58.5M 2992M 17.2.0 e1d6a67b021e 1d9e6609b585 2024-05-06T13:55:08.243 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (25m) 13s ago 25m 52.4M 2992M 17.2.0 e1d6a67b021e 37a61d295111 2024-05-06T13:55:08.243 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (25m) 13s ago 25m 52.9M 2992M 17.2.0 e1d6a67b021e 8feb7c077aaa 2024-05-06T13:55:08.243 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (24m) 13s ago 24m 58.1M 3248M 17.2.0 e1d6a67b021e f865acc44a06 2024-05-06T13:55:08.243 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (24m) 13s ago 24m 52.5M 3248M 17.2.0 e1d6a67b021e a55bd4122f6f 2024-05-06T13:55:08.243 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (24m) 13s ago 24m 51.2M 3248M 17.2.0 e1d6a67b021e 362938a0b357 2024-05-06T13:55:08.243 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (23m) 13s ago 23m 53.7M 3248M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T13:55:08.243 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (18s) 13s ago 23m 45.9M - 2.43.0 a07b618ecd1d 10d5c513bb9b 2024-05-06T13:55:08.243 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (21m) 13s ago 21m 77.2M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T13:55:08.243 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (21m) 13s ago 21m 76.1M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T13:55:08.641 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:55:08.641 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-05-06T13:55:08.641 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-05-06T13:55:08.641 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:55:08.641 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-05-06T13:55:08.641 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T13:55:08.641 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:55:08.641 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-05-06T13:55:08.641 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-06T13:55:08.641 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:55:08.641 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": {}, 2024-05-06T13:55:08.641 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw": { 2024-05-06T13:55:08.642 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T13:55:08.642 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:55:08.642 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-05-06T13:55:08.642 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-05-06T13:55:08.642 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T13:55:08.642 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T13:55:08.642 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:55:08.957 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:08 smithi077 ceph-mon[22428]: from='client.15726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:55:08.957 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:08 smithi077 ceph-mon[22428]: from='client.25633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:55:08.957 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:08 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/429286637' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:08.957 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:08 smithi077 ceph-mon[27166]: from='client.15726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:55:08.958 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:08 smithi077 ceph-mon[27166]: from='client.25633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:55:08.958 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:08 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/429286637' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:08.979 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:55:08.980 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T13:55:08.980 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-05-06T13:55:08.980 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-05-06T13:55:08.980 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [ 2024-05-06T13:55:08.980 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr" 2024-05-06T13:55:08.980 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-05-06T13:55:08.980 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "2/2 daemons upgraded", 2024-05-06T13:55:08.980 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "Currently upgrading alertmanager daemons", 2024-05-06T13:55:08.980 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-05-06T13:55:08.980 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:55:09.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:08 smithi170 ceph-mon[23272]: from='client.15726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:55:09.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:08 smithi170 ceph-mon[23272]: from='client.25633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:55:09.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:08 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/429286637' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:10.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:09 smithi077 ceph-mon[22428]: from='client.25639 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:55:10.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:09 smithi077 ceph-mon[22428]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:10.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:09 smithi077 ceph-mon[27166]: from='client.25639 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:55:10.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:09 smithi077 ceph-mon[27166]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:10.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:09 smithi170 ceph-mon[23272]: from='client.25639 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:55:10.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:09 smithi170 ceph-mon[23272]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:10.897 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:10 smithi077 ceph-mon[22428]: from='client.15750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:55:10.898 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:10 smithi077 ceph-mon[27166]: from='client.15750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:55:10.898 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:10 smithi077 systemd[1]: Stopping Ceph alertmanager.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:55:11.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:10 smithi170 ceph-mon[23272]: from='client.15750 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:55:11.257 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:10 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[56677]: level=info ts=2024-05-06T13:55:10.897Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-05-06T13:55:11.257 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:10 smithi077 podman[79282]: 2024-05-06 13:55:10.948454423 +0000 UTC m=+0.166285299 container died 0f855548777f89ed44bceeee061f31f3b5a40bcfc1e91dd8df0d4cbbdb01940e (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T13:55:11.552 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:11 smithi077 podman[79282]: 2024-05-06 13:55:11.440083336 +0000 UTC m=+0.657914288 container cleanup 0f855548777f89ed44bceeee061f31f3b5a40bcfc1e91dd8df0d4cbbdb01940e (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T13:55:11.552 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:11 smithi077 bash[79282]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a 2024-05-06T13:55:11.806 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:11 smithi077 podman[79297]: 2024-05-06 13:55:11.749961725 +0000 UTC m=+0.793784838 container remove 0f855548777f89ed44bceeee061f31f3b5a40bcfc1e91dd8df0d4cbbdb01940e (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T13:55:12.008 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:11 smithi077 ceph-mon[22428]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:12.008 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:11 smithi077 podman[79297]: 2024-05-06 13:55:11.850921384 +0000 UTC m=+0.894744496 volume remove b6479ecfa0b406387bf58df1a23204dd25c00c04a98d58109ab829983e3507a4 2024-05-06T13:55:12.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:11 smithi077 ceph-mon[27166]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:12.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:11 smithi170 ceph-mon[23272]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:12.302 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:12 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-05-06T13:55:12.303 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:12 smithi077 systemd[1]: Stopped Ceph alertmanager.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:55:12.303 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:12 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@alertmanager.a.service: Consumed 1.528s CPU time. 2024-05-06T13:55:12.303 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:12 smithi077 systemd[1]: Starting Ceph alertmanager.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:55:12.728 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:12 smithi077 podman[79428]: 2024-05-06 13:55:12.339863648 +0000 UTC m=+0.017951685 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-05-06T13:55:12.728 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:12 smithi077 podman[79428]: 2024-05-06 13:55:12.728560583 +0000 UTC m=+0.406648650 volume create 96fe4f4a20341b2da21ba624b7eec8d638433ed0e509b5d2c29d85764d10cd33 2024-05-06T13:55:12.994 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:12 smithi077 podman[79428]: 2024-05-06 13:55:12.803959349 +0000 UTC m=+0.482047371 container create 997175e3e64efa0dd530a32eda4af713c04a08ecf93df0f3ab8d801e0741ab5e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T13:55:13.245 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:13 smithi077 podman[79428]: 2024-05-06 13:55:13.132172056 +0000 UTC m=+0.810260078 container init 997175e3e64efa0dd530a32eda4af713c04a08ecf93df0f3ab8d801e0741ab5e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T13:55:13.245 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:13 smithi077 podman[79428]: 2024-05-06 13:55:13.135037473 +0000 UTC m=+0.813125496 container start 997175e3e64efa0dd530a32eda4af713c04a08ecf93df0f3ab8d801e0741ab5e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T13:55:13.245 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[79442]: ts=2024-05-06T13:55:13.148Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-05-06T13:55:13.245 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[79442]: ts=2024-05-06T13:55:13.148Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-05-06T13:55:13.245 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[79442]: ts=2024-05-06T13:55:13.150Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.77 port=9094 2024-05-06T13:55:13.245 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[79442]: ts=2024-05-06T13:55:13.152Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-05-06T13:55:13.245 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[79442]: ts=2024-05-06T13:55:13.176Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-05-06T13:55:13.245 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[79442]: ts=2024-05-06T13:55:13.176Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-05-06T13:55:13.246 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[79442]: ts=2024-05-06T13:55:13.179Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-05-06T13:55:13.246 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[79442]: ts=2024-05-06T13:55:13.179Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-05-06T13:55:13.246 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:13 smithi077 bash[79428]: 997175e3e64efa0dd530a32eda4af713c04a08ecf93df0f3ab8d801e0741ab5e 2024-05-06T13:55:13.552 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:13 smithi077 systemd[1]: Started Ceph alertmanager.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:55:14.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:55:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:55:13] "GET /metrics HTTP/1.1" 200 37986 "" "Prometheus/2.43.0" 2024-05-06T13:55:14.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:13 smithi077 ceph-mon[22428]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:14.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:13 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:14.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:13 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:14.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:13 smithi077 ceph-mon[27166]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:14.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:13 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:14.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:13 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:14.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:13 smithi170 ceph-mon[23272]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:14.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:13 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:14.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:13 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:15.552 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:15 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[79442]: ts=2024-05-06T13:55:15.153Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000575475s 2024-05-06T13:55:15.891 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:15 smithi077 ceph-mon[22428]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:55:15.892 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:15 smithi077 ceph-mon[22428]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:15.892 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:15 smithi077 ceph-mon[27166]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:55:15.892 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:15 smithi077 ceph-mon[27166]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:16.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:15 smithi170 ceph-mon[23272]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:55:16.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:15 smithi170 ceph-mon[23272]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:18.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:17 smithi170 ceph-mon[23272]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:18.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:17 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:18.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:17 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:18.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:17 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:18.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:17 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:18.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:17 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:55:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:17 smithi077 ceph-mon[22428]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:17 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:17 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:17 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:18.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:17 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:18.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:17 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:55:18.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:17 smithi077 ceph-mon[27166]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:18.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:17 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:18.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:17 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:18.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:17 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:18.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:17 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:18.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:17 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:55:20.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:19 smithi077 ceph-mon[22428]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:20.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:19 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:20.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:19 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:20.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:19 smithi077 ceph-mon[27166]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:20.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:19 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:20.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:19 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:20.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:19 smithi170 ceph-mon[23272]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:20.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:19 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:20.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:19 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:21.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:21 smithi077 ceph-mon[22428]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:21.995 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:21 smithi077 ceph-mon[27166]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:22.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:21 smithi170 ceph-mon[23272]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:22.754 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:55:22 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:55:22] ENGINE Bus STOPPING 2024-05-06T13:55:22.754 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:55:22 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:55:22] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-06T13:55:22.754 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:55:22 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:55:22] ENGINE Bus STOPPED 2024-05-06T13:55:22.754 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:55:22 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:55:22] ENGINE Bus STARTING 2024-05-06T13:55:22.754 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:55:22 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:55:22] ENGINE Serving on http://:::9283 2024-05-06T13:55:22.755 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:55:22 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:55:22] ENGINE Bus STARTED 2024-05-06T13:55:23.476 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 13:55:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[79442]: ts=2024-05-06T13:55:23.154Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.001821067s 2024-05-06T13:55:23.531 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:23 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:23.531 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:23 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:23.531 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:23 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:55:23.531 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:23 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:55:23.531 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:23 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:23.531 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:23 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T13:55:23.531 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:23 smithi170 ceph-mon[23272]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T13:55:23.532 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:23 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi077:9093"}]: dispatch 2024-05-06T13:55:23.532 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:23 smithi170 ceph-mon[23272]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi077:9093"}]: dispatch 2024-05-06T13:55:23.532 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:23 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:23.532 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:23 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:55:23.532 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:23 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.532 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:23 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.532 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:23 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.532 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:23 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.532 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:23 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.532 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:23 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.532 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:23 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.532 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:23 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.532 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:23 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.533 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:23 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.533 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:23 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.533 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:23 smithi170 ceph-mon[23272]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:23.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:23.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:23.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:55:23.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:55:23.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:23.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T13:55:23.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[27166]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T13:55:23.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi077:9093"}]: dispatch 2024-05-06T13:55:23.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[27166]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi077:9093"}]: dispatch 2024-05-06T13:55:23.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:23.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:55:23.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[27166]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:23.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:23.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:23.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:55:23.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:55:23.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:23.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T13:55:23.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[22428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-05-06T13:55:23.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi077:9093"}]: dispatch 2024-05-06T13:55:23.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[22428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi077:9093"}]: dispatch 2024-05-06T13:55:23.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:23.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:55:23.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.805 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.806 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.806 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.806 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.806 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.806 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.806 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.806 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.806 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.806 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:23.806 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:23 smithi077 ceph-mon[22428]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:24.302 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:55:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:55:23] "GET /metrics HTTP/1.1" 200 37981 "" "Prometheus/2.43.0" 2024-05-06T13:55:26.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:25 smithi077 ceph-mon[27166]: Upgrade: Updating grafana.a 2024-05-06T13:55:26.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:25 smithi077 ceph-mon[27166]: Deploying daemon grafana.a on smithi170 2024-05-06T13:55:26.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:25 smithi077 ceph-mon[27166]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:55:26.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:25 smithi077 ceph-mon[27166]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:26.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:25 smithi077 ceph-mon[22428]: Upgrade: Updating grafana.a 2024-05-06T13:55:26.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:25 smithi077 ceph-mon[22428]: Deploying daemon grafana.a on smithi170 2024-05-06T13:55:26.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:25 smithi077 ceph-mon[22428]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:55:26.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:25 smithi077 ceph-mon[22428]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:26.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:25 smithi170 ceph-mon[23272]: Upgrade: Updating grafana.a 2024-05-06T13:55:26.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:25 smithi170 ceph-mon[23272]: Deploying daemon grafana.a on smithi170 2024-05-06T13:55:26.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:25 smithi170 ceph-mon[23272]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:55:26.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:25 smithi170 ceph-mon[23272]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:28.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:27 smithi077 ceph-mon[22428]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:28.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:27 smithi077 ceph-mon[27166]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:28.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:27 smithi170 ceph-mon[23272]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:30.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:29 smithi077 ceph-mon[22428]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:30.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:29 smithi077 ceph-mon[27166]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:30.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:29 smithi170 ceph-mon[23272]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:32.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:31 smithi077 ceph-mon[22428]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:32.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:31 smithi077 ceph-mon[27166]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:32.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:31 smithi170 ceph-mon[23272]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:33.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:32 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:55:33.099 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:32 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:55:33.099 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:32 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:55:33.807 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:33 smithi170 systemd[1]: Stopping Ceph grafana.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:55:33.997 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:33 smithi077 ceph-mon[22428]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:33.998 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:33 smithi077 ceph-mon[27166]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:33.998 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:55:33 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:55:33] "GET /metrics HTTP/1.1" 200 37983 "" "Prometheus/2.43.0" 2024-05-06T13:55:34.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:33 smithi170 ceph-mon[23272]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:34.077 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:33 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[47195]: t=2024-05-06T13:55:33+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-05-06T13:55:34.077 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:33 smithi170 podman[64087]: 2024-05-06 13:55:33.957546354 +0000 UTC m=+0.136415432 container died fdcecac7992557cf2777d9450df344219792984e400651bbc54db2739786f0b8 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a, description=Ceph Grafana Container, distribution-scope=public, io.buildah.version=1.24.2, com.redhat.component=ubi8-container, vcs-type=git, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, name=ubi8, version=8.5, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner ) 2024-05-06T13:55:34.574 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:34 smithi170 podman[64087]: 2024-05-06 13:55:34.348043177 +0000 UTC m=+0.526912263 container cleanup fdcecac7992557cf2777d9450df344219792984e400651bbc54db2739786f0b8 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, io.openshift.tags=base rhel8, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , vcs-type=git, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, architecture=x86_64, description=Ceph Grafana Container, release=236.1648460182, io.buildah.version=1.24.2) 2024-05-06T13:55:34.574 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:34 smithi170 bash[64087]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a 2024-05-06T13:55:34.826 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:34 smithi170 podman[64101]: 2024-05-06 13:55:34.57406657 +0000 UTC m=+0.618955048 container remove fdcecac7992557cf2777d9450df344219792984e400651bbc54db2739786f0b8 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a, version=8.5, com.redhat.component=ubi8-container, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, vendor=Red Hat, Inc., io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, vcs-type=git, architecture=x86_64, maintainer=Paul Cuzner , release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-05-06T13:55:34.826 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:34 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@grafana.a.service: Deactivated successfully. 2024-05-06T13:55:34.827 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:34 smithi170 systemd[1]: Stopped Ceph grafana.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:55:34.827 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:34 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@grafana.a.service: Consumed 5.545s CPU time. 2024-05-06T13:55:35.239 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:34 smithi170 systemd[1]: Starting Ceph grafana.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:55:35.239 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 podman[64233]: 2024-05-06 13:55:35.005976203 +0000 UTC m=+0.018536805 image pull 2bacad6d85d802cd176c146528adcc0e47a8a83e94bc4a8b7ba1a9b9ef2c65e7 quay.io/ceph/grafana:9.4.12 2024-05-06T13:55:35.239 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 podman[64233]: 2024-05-06 13:55:35.133524521 +0000 UTC m=+0.146085108 container create 140bb9a2760a2fa6f9beba2b09b5ac6feda1b11d3acb11aaf35ee8a48a326501 (image=quay.io/ceph/grafana:9.4.12, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-05-06T13:55:35.517 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 podman[64233]: 2024-05-06 13:55:35.411012244 +0000 UTC m=+0.423572831 container init 140bb9a2760a2fa6f9beba2b09b5ac6feda1b11d3acb11aaf35ee8a48a326501 (image=quay.io/ceph/grafana:9.4.12, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-05-06T13:55:35.517 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 podman[64233]: 2024-05-06 13:55:35.413762887 +0000 UTC m=+0.426323474 container start 140bb9a2760a2fa6f9beba2b09b5ac6feda1b11d3acb11aaf35ee8a48a326501 (image=quay.io/ceph/grafana:9.4.12, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-05-06T13:55:35.810 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 bash[64233]: 140bb9a2760a2fa6f9beba2b09b5ac6feda1b11d3acb11aaf35ee8a48a326501 2024-05-06T13:55:35.811 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 systemd[1]: Started Ceph grafana.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:55:35.811 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=settings t=2024-05-06T13:55:35.566206702Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-05-06T13:55:35.811 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=settings t=2024-05-06T13:55:35.566456498Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-05-06T13:55:35.811 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=settings t=2024-05-06T13:55:35.566470816Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-05-06T13:55:35.811 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=settings t=2024-05-06T13:55:35.566476485Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-05-06T13:55:35.811 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=settings t=2024-05-06T13:55:35.566481056Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-05-06T13:55:35.811 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=settings t=2024-05-06T13:55:35.566485491Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-05-06T13:55:35.811 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=settings t=2024-05-06T13:55:35.566489894Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-05-06T13:55:35.811 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=settings t=2024-05-06T13:55:35.566494392Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-05-06T13:55:35.811 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=settings t=2024-05-06T13:55:35.566498931Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-05-06T13:55:35.811 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=settings t=2024-05-06T13:55:35.566503871Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-05-06T13:55:35.812 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=settings t=2024-05-06T13:55:35.56650876Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-05-06T13:55:35.812 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=settings t=2024-05-06T13:55:35.566513314Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-05-06T13:55:35.812 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=settings t=2024-05-06T13:55:35.56651787Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-05-06T13:55:35.812 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=settings t=2024-05-06T13:55:35.566522394Z level=info msg="Path Home" path=/usr/share/grafana 2024-05-06T13:55:35.812 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=settings t=2024-05-06T13:55:35.566526837Z level=info msg="Path Data" path=/var/lib/grafana 2024-05-06T13:55:35.812 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=settings t=2024-05-06T13:55:35.56653114Z level=info msg="Path Logs" path=/var/log/grafana 2024-05-06T13:55:35.812 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=settings t=2024-05-06T13:55:35.566535422Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-05-06T13:55:35.813 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=settings t=2024-05-06T13:55:35.566539795Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-05-06T13:55:35.813 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=settings t=2024-05-06T13:55:35.566544294Z level=info msg="App mode production" 2024-05-06T13:55:35.814 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=sqlstore t=2024-05-06T13:55:35.566600261Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-05-06T13:55:35.814 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=sqlstore t=2024-05-06T13:55:35.566615881Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-05-06T13:55:35.814 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.578119747Z level=info msg="Starting DB migrations" 2024-05-06T13:55:35.814 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.581559828Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-05-06T13:55:35.814 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.592926919Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-05-06T13:55:35.814 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.594198164Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-05-06T13:55:35.814 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.594489729Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-05-06T13:55:35.814 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.595649701Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-05-06T13:55:35.814 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.59671899Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-05-06T13:55:35.814 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.597746506Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-05-06T13:55:35.814 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.598047728Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-05-06T13:55:35.814 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.598514663Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-05-06T13:55:35.815 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.599033608Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-05-06T13:55:35.815 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.59934779Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-05-06T13:55:35.815 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.600399307Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-05-06T13:55:35.815 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.600719372Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-05-06T13:55:35.815 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.601764917Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-05-06T13:55:35.815 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.602823673Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-05-06T13:55:35.815 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.603877118Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-05-06T13:55:35.815 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.604184714Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-05-06T13:55:35.815 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.605217552Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-05-06T13:55:35.815 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.606264157Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-05-06T13:55:35.815 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.606568611Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-05-06T13:55:35.815 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.607647868Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-05-06T13:55:35.815 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.608716681Z level=info msg="Executing migration" id="create provenance_type table" 2024-05-06T13:55:35.816 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.609164486Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-05-06T13:55:35.816 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.609604256Z level=info msg="Executing migration" id="create alert_image table" 2024-05-06T13:55:35.816 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.610040865Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-05-06T13:55:35.816 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.610471687Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-05-06T13:55:35.816 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.610716027Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-05-06T13:55:35.816 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.611114903Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-05-06T13:55:35.816 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.611544817Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-05-06T13:55:35.816 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.611628359Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-05-06T13:55:35.816 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.611838112Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-05-06T13:55:35.816 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.612304925Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-05-06T13:55:35.816 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.612734312Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-05-06T13:55:35.816 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.612968061Z level=info msg="Executing migration" id="create secrets table" 2024-05-06T13:55:35.816 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.613382806Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-05-06T13:55:35.817 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.619448068Z level=info msg="Executing migration" id="add name column into data_keys" 2024-05-06T13:55:35.817 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.620564672Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-05-06T13:55:35.817 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.620845106Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-05-06T13:55:35.817 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.627135363Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-05-06T13:55:35.817 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.633495318Z level=info msg="Executing migration" id="add column hidden to role table" 2024-05-06T13:55:35.817 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.634656903Z level=info msg="Executing migration" id="create query_history table v1" 2024-05-06T13:55:35.817 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.635035589Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-05-06T13:55:35.817 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.635520885Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-05-06T13:55:35.817 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.635756704Z level=info msg="Executing migration" id="teams permissions migration" 2024-05-06T13:55:35.817 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.636163202Z level=info msg="Executing migration" id="dashboard permissions" 2024-05-06T13:55:35.817 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.63748083Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-05-06T13:55:35.817 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.638142189Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-05-06T13:55:35.817 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.63842192Z level=info msg="Executing migration" id="alerting notification permissions" 2024-05-06T13:55:35.818 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.638719469Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-05-06T13:55:35.818 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.639089814Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-05-06T13:55:35.818 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.639515616Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-05-06T13:55:35.818 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.640634571Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-05-06T13:55:35.818 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.640917419Z level=info msg="Executing migration" id="create correlation table v1" 2024-05-06T13:55:35.818 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.641348566Z level=info msg="Executing migration" id="add index correlations.uid" 2024-05-06T13:55:35.818 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.641784839Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-05-06T13:55:35.818 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.642220695Z level=info msg="Executing migration" id="add correlation config column" 2024-05-06T13:55:35.818 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.643307004Z level=info msg="Executing migration" id="create entity_events table" 2024-05-06T13:55:35.818 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.643685175Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-05-06T13:55:35.818 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.644088018Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-05-06T13:55:35.818 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.644189626Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-05-06T13:55:35.819 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.644466439Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-05-06T13:55:35.819 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.644541435Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-05-06T13:55:35.819 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.644798022Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-05-06T13:55:35.819 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.645198552Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-05-06T13:55:35.819 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.645570412Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-05-06T13:55:35.819 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.645994266Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-05-06T13:55:35.819 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.646413327Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-05-06T13:55:35.819 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.646773857Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-05-06T13:55:35.819 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.647197699Z level=info msg="Executing migration" id="Drop public config table" 2024-05-06T13:55:35.819 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.647599845Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-05-06T13:55:35.819 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.64797056Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-05-06T13:55:35.819 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.648427859Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-05-06T13:55:35.819 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.648809207Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-05-06T13:55:35.820 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.649232891Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-05-06T13:55:35.820 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.653091704Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-05-06T13:55:35.820 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.654214587Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-05-06T13:55:35.820 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.655473682Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-05-06T13:55:35.820 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.655773347Z level=info msg="Executing migration" id="add share column" 2024-05-06T13:55:35.820 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.656924461Z level=info msg="Executing migration" id="create default alerting folders" 2024-05-06T13:55:35.820 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.657405631Z level=info msg="Executing migration" id="create file table" 2024-05-06T13:55:35.820 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.657782444Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-05-06T13:55:35.820 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.658222517Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-05-06T13:55:35.820 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.65863329Z level=info msg="Executing migration" id="create file_meta table" 2024-05-06T13:55:35.820 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.658988741Z level=info msg="Executing migration" id="file table idx: path key" 2024-05-06T13:55:35.820 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.659428983Z level=info msg="Executing migration" id="set path collation in file table" 2024-05-06T13:55:35.820 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.659662232Z level=info msg="Executing migration" id="managed permissions migration" 2024-05-06T13:55:35.820 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.660876542Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-05-06T13:55:35.821 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.661580413Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-05-06T13:55:35.821 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.662170574Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-05-06T13:55:35.821 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.663349126Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-05-06T13:55:35.821 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.663629843Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-05-06T13:55:35.821 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.66406578Z level=info msg="Executing migration" id="update group index for alert rules" 2024-05-06T13:55:35.821 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.664430935Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-05-06T13:55:35.821 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.664899968Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-05-06T13:55:35.821 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.665253756Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-05-06T13:55:35.821 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.666436364Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-05-06T13:55:35.821 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.667605412Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-05-06T13:55:35.821 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.668042334Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-05-06T13:55:35.821 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.681602361Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-05-06T13:55:35.821 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.682073358Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-05-06T13:55:35.822 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.682497029Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-05-06T13:55:35.822 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.686871713Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-05-06T13:55:35.822 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.687368805Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-05-06T13:55:35.822 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.687681417Z level=info msg="Executing migration" id="create folder table" 2024-05-06T13:55:35.822 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.688031403Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-05-06T13:55:35.822 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.688447734Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-05-06T13:55:35.822 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.688850255Z level=info msg="Executing migration" id="Update folder title length" 2024-05-06T13:55:35.822 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.689118233Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-05-06T13:55:35.822 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=migrator t=2024-05-06T13:55:35.689563695Z level=info msg="migrations completed" performed=105 skipped=376 duration=108.044442ms 2024-05-06T13:55:35.822 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=sqlstore t=2024-05-06T13:55:35.690237796Z level=info msg="Created default organization" 2024-05-06T13:55:35.822 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=plugin.finder t=2024-05-06T13:55:35.722969048Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-05-06T13:55:35.822 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=secrets t=2024-05-06T13:55:35.723110998Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-05-06T13:55:35.823 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=query_data t=2024-05-06T13:55:35.725221322Z level=info msg="Query Service initialization" 2024-05-06T13:55:35.823 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=live.push_http t=2024-05-06T13:55:35.727704924Z level=info msg="Live Push Gateway initialization" 2024-05-06T13:55:36.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:35 smithi170 ceph-mon[23272]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:55:36.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:35 smithi170 ceph-mon[23272]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:36.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:35 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:36.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:35 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:36.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:35 smithi077 ceph-mon[22428]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:55:36.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:35 smithi077 ceph-mon[22428]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:36.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:35 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:36.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:35 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:36.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:35 smithi077 ceph-mon[27166]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:55:36.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:35 smithi077 ceph-mon[27166]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:36.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:35 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:36.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:35 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:36.683 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:36 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=infra.usagestats.collector t=2024-05-06T13:55:36.386310869Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-05-06T13:55:36.684 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:36 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=provisioning.datasources t=2024-05-06T13:55:36.388743748Z level=info msg="deleted correlations based on configuration" ds_name=Dashboard1 2024-05-06T13:55:36.684 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:36 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=provisioning.datasources t=2024-05-06T13:55:36.388810509Z level=info msg="deleted datasource based on configuration" name=Dashboard1 2024-05-06T13:55:36.684 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:36 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=provisioning.datasources t=2024-05-06T13:55:36.389003687Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-05-06T13:55:36.684 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:36 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=provisioning.alerting t=2024-05-06T13:55:36.41315387Z level=info msg="starting to provision alerting" 2024-05-06T13:55:36.684 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:36 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=provisioning.alerting t=2024-05-06T13:55:36.413173359Z level=info msg="finished to provision alerting" 2024-05-06T13:55:36.684 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:36 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=grafanaStorageLogger t=2024-05-06T13:55:36.413354359Z level=info msg="storage starting" 2024-05-06T13:55:36.684 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:36 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=ngalert.state.manager t=2024-05-06T13:55:36.41360202Z level=info msg="Warming state cache for startup" 2024-05-06T13:55:36.684 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:36 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=http.server t=2024-05-06T13:55:36.415196856Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-05-06T13:55:36.684 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:36 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=ngalert.state.manager t=2024-05-06T13:55:36.425001171Z level=info msg="State cache has been initialized" states=0 duration=11.398222ms 2024-05-06T13:55:36.684 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:36 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=ngalert.multiorg.alertmanager t=2024-05-06T13:55:36.425031719Z level=info msg="starting MultiOrg Alertmanager" 2024-05-06T13:55:36.684 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:36 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=ticker t=2024-05-06T13:55:36.425065379Z level=info msg=starting first_tick=2024-05-06T13:55:40Z 2024-05-06T13:55:36.684 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:36 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=sqlstore.transactions t=2024-05-06T13:55:36.437390166Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-05-06T13:55:36.684 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:36 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=sqlstore.transactions t=2024-05-06T13:55:36.44790405Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-05-06T13:55:36.685 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 13:55:36 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=sqlstore.transactions t=2024-05-06T13:55:36.459588392Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-05-06T13:55:38.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:37 smithi170 ceph-mon[23272]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:38.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:37 smithi077 ceph-mon[22428]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:38.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:37 smithi077 ceph-mon[27166]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:39.340 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T13:55:39.986 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T13:55:39.986 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (27s) 22s ago 22m 17.1M - 0.25.0 c8568f914cd2 997175e3e64e 2024-05-06T13:55:39.986 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 starting - - - - 2024-05-06T13:55:39.986 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (66s) 22s ago 21m 52.4M - 3.5 e1d6a67b021e 9353a303fb55 2024-05-06T13:55:39.986 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283,8765 running (107s) 22s ago 27m 451M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 125521be53a3 2024-05-06T13:55:39.987 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (11m) 22s ago 28m 504M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T13:55:39.987 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (28m) 22s ago 29m 78.0M 2048M 17.2.0 e1d6a67b021e ac46f91694f8 2024-05-06T13:55:39.987 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (27m) 22s ago 27m 56.0M 2048M 17.2.0 e1d6a67b021e 24be6fce9d28 2024-05-06T13:55:39.987 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (27m) 22s ago 27m 57.2M 2048M 17.2.0 e1d6a67b021e 83af336ef762 2024-05-06T13:55:39.987 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (87s) 22s ago 23m 14.4M - 1.5.0 0da6a335fe13 67f45fa4968b 2024-05-06T13:55:39.987 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (79s) 22s ago 23m 16.9M - 1.5.0 0da6a335fe13 02081b71a5dd 2024-05-06T13:55:39.987 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (27m) 22s ago 27m 54.4M 2992M 17.2.0 e1d6a67b021e 2ccee2bb6e96 2024-05-06T13:55:39.987 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (26m) 22s ago 26m 58.5M 2992M 17.2.0 e1d6a67b021e 1d9e6609b585 2024-05-06T13:55:39.987 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (26m) 22s ago 26m 52.4M 2992M 17.2.0 e1d6a67b021e 37a61d295111 2024-05-06T13:55:39.987 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (25m) 22s ago 25m 52.9M 2992M 17.2.0 e1d6a67b021e 8feb7c077aaa 2024-05-06T13:55:39.987 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (25m) 22s ago 25m 58.1M 3248M 17.2.0 e1d6a67b021e f865acc44a06 2024-05-06T13:55:39.987 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (25m) 22s ago 25m 52.5M 3248M 17.2.0 e1d6a67b021e a55bd4122f6f 2024-05-06T13:55:39.987 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (24m) 22s ago 24m 51.2M 3248M 17.2.0 e1d6a67b021e 362938a0b357 2024-05-06T13:55:39.987 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (24m) 22s ago 24m 53.7M 3248M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T13:55:39.987 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (49s) 22s ago 23m 46.8M - 2.43.0 a07b618ecd1d 10d5c513bb9b 2024-05-06T13:55:39.988 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (22m) 22s ago 22m 77.3M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T13:55:39.988 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (22m) 22s ago 22m 76.2M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T13:55:40.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:39 smithi170 ceph-mon[23272]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:40.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:39 smithi077 ceph-mon[22428]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:40.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:39 smithi077 ceph-mon[27166]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:40.387 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:55:40.387 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-05-06T13:55:40.387 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-05-06T13:55:40.387 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:55:40.387 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-05-06T13:55:40.387 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T13:55:40.387 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:55:40.387 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-05-06T13:55:40.387 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-06T13:55:40.388 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:55:40.388 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": {}, 2024-05-06T13:55:40.388 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw": { 2024-05-06T13:55:40.388 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T13:55:40.388 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:55:40.388 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-05-06T13:55:40.388 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-05-06T13:55:40.388 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T13:55:40.388 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T13:55:40.388 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:55:40.719 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:55:40.719 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T13:55:40.719 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-05-06T13:55:40.719 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-05-06T13:55:40.719 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [ 2024-05-06T13:55:40.720 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr" 2024-05-06T13:55:40.720 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-05-06T13:55:40.720 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "2/2 daemons upgraded", 2024-05-06T13:55:40.720 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "Currently upgrading grafana daemons", 2024-05-06T13:55:40.720 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-05-06T13:55:40.720 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:55:41.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:40 smithi077 ceph-mon[22428]: from='client.25657 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:55:41.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:40 smithi077 ceph-mon[22428]: from='client.25663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:55:41.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:40 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:41.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:40 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:41.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:40 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2692943284' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:41.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:40 smithi077 ceph-mon[27166]: from='client.25657 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:55:41.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:40 smithi077 ceph-mon[27166]: from='client.25663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:55:41.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:40 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:41.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:40 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:41.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:40 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2692943284' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:41.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:40 smithi170 ceph-mon[23272]: from='client.25657 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:55:41.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:40 smithi170 ceph-mon[23272]: from='client.25663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:55:41.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:40 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:41.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:40 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:41.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:40 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2692943284' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:41 smithi170 ceph-mon[23272]: from='client.15768 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:55:42.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:41 smithi170 ceph-mon[23272]: from='client.25681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:55:42.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:41 smithi170 ceph-mon[23272]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:42.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:41 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:42.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:41 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:42.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:41 smithi077 ceph-mon[27166]: from='client.15768 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:55:42.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:41 smithi077 ceph-mon[27166]: from='client.25681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:55:42.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:41 smithi077 ceph-mon[27166]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:42.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:41 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:42.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:41 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:42.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:41 smithi077 ceph-mon[22428]: from='client.15768 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:55:42.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:41 smithi077 ceph-mon[22428]: from='client.25681 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:55:42.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:41 smithi077 ceph-mon[22428]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:42.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:41 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:42.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:41 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:43.802 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:55:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:55:43] "GET /metrics HTTP/1.1" 200 37983 "" "Prometheus/2.43.0" 2024-05-06T13:55:44.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:43 smithi170 ceph-mon[23272]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:44.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:43 smithi077 ceph-mon[22428]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:44.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:43 smithi077 ceph-mon[27166]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:45.302 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:55:44 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:55:44] ENGINE Bus STOPPING 2024-05-06T13:55:45.303 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:55:44 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:55:44] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-06T13:55:45.303 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:55:44 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:55:44] ENGINE Bus STOPPED 2024-05-06T13:55:45.303 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:55:44 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:55:44] ENGINE Bus STARTING 2024-05-06T13:55:45.303 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:55:45 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:55:45] ENGINE Serving on http://:::9283 2024-05-06T13:55:45.303 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:55:45 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:55:45] ENGINE Bus STARTED 2024-05-06T13:55:46.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:55:46.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:46.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:46.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:46.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:55:46.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:55:46.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:46.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T13:55:46.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T13:55:46.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi170.front.sepia.ceph.com:3000"}]: dispatch 2024-05-06T13:55:46.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi170.front.sepia.ceph.com:3000"}]: dispatch 2024-05-06T13:55:46.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:46.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:55:46.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: Upgrade: Finalizing container_image settings 2024-05-06T13:55:46.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:46.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:55:46.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:46.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:46.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:46.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:55:46.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:55:46.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:46.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T13:55:46.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T13:55:46.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi170.front.sepia.ceph.com:3000"}]: dispatch 2024-05-06T13:55:46.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi170.front.sepia.ceph.com:3000"}]: dispatch 2024-05-06T13:55:46.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:46.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:55:46.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: Upgrade: Finalizing container_image settings 2024-05-06T13:55:46.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:46.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-05-06T13:55:46.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-05-06T13:55:46.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:55:46.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-05-06T13:55:46.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-05-06T13:55:46.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-05-06T13:55:46.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-05-06T13:55:46.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-05-06T13:55:46.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-05-06T13:55:46.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-05-06T13:55:46.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-05-06T13:55:46.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-05-06T13:55:46.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:55:46.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-05-06T13:55:46.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-05-06T13:55:46.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-05-06T13:55:46.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-05-06T13:55:46.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-05-06T13:55:46.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-05-06T13:55:46.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-05-06T13:55:46.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:55:46.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:55:46.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:55:46.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:55:46.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:55:46.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:55:46.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-05-06T13:55:46.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-05-06T13:55:46.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:55:46.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:55:46.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:46.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:55:46.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:55:46.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:46.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-05-06T13:55:46.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-05-06T13:55:46.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:55:46.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-05-06T13:55:46.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-05-06T13:55:46.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-05-06T13:55:46.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-05-06T13:55:46.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-05-06T13:55:46.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-05-06T13:55:46.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-05-06T13:55:46.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-05-06T13:55:46.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-05-06T13:55:46.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:55:46.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-05-06T13:55:46.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-05-06T13:55:46.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-05-06T13:55:46.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-05-06T13:55:46.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-05-06T13:55:46.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-05-06T13:55:46.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-05-06T13:55:46.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:55:46.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:55:46.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:55:46.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:55:46.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:55:46.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:55:46.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-05-06T13:55:46.063 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-05-06T13:55:46.063 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:55:46.063 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:55:46.063 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:46.063 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:55:46.063 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:55:46.063 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:45 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:46.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:55:46.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:46.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:46.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:46.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:55:46.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:55:46.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:46.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T13:55:46.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-05-06T13:55:46.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi170.front.sepia.ceph.com:3000"}]: dispatch 2024-05-06T13:55:46.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi170.front.sepia.ceph.com:3000"}]: dispatch 2024-05-06T13:55:46.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:46.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:55:46.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:55:46.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: Upgrade: Finalizing container_image settings 2024-05-06T13:55:46.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:46.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-05-06T13:55:46.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-05-06T13:55:46.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:55:46.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-05-06T13:55:46.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-05-06T13:55:46.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-05-06T13:55:46.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-05-06T13:55:46.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-05-06T13:55:46.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-05-06T13:55:46.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-05-06T13:55:46.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-05-06T13:55:46.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-05-06T13:55:46.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:55:46.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-05-06T13:55:46.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-05-06T13:55:46.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-05-06T13:55:46.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-05-06T13:55:46.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-05-06T13:55:46.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-05-06T13:55:46.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-05-06T13:55:46.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:55:46.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:55:46.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:55:46.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:55:46.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:55:46.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:55:46.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-05-06T13:55:46.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-05-06T13:55:46.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:55:46.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:55:46.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:46.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:55:46.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:55:46.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:45 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:47.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:46 smithi077 ceph-mon[22428]: Upgrade: Complete! 2024-05-06T13:55:47.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:46 smithi077 ceph-mon[27166]: Upgrade: Complete! 2024-05-06T13:55:47.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:46 smithi170 ceph-mon[23272]: Upgrade: Complete! 2024-05-06T13:55:48.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:47 smithi077 ceph-mon[22428]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:48.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:47 smithi077 ceph-mon[27166]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:48.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:47 smithi170 ceph-mon[23272]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:49.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:48 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:55:49.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:48 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:49.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:48 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:55:49.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:48 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:49.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:48 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:55:49.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:48 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:55:50.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:49 smithi170 ceph-mon[23272]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:50.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:49 smithi077 ceph-mon[22428]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:50.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:49 smithi077 ceph-mon[27166]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:52.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:51 smithi170 ceph-mon[23272]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:52.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:51 smithi077 ceph-mon[27166]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:52.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:51 smithi077 ceph-mon[22428]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:54.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:53 smithi077 ceph-mon[22428]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:54.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:53 smithi077 ceph-mon[27166]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:54.053 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:55:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:55:53] "GET /metrics HTTP/1.1" 200 37993 "" "Prometheus/2.43.0" 2024-05-06T13:55:54.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:53 smithi170 ceph-mon[23272]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:56.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:55 smithi170 ceph-mon[23272]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:55:56.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:55 smithi170 ceph-mon[23272]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:56.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:55 smithi077 ceph-mon[22428]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:55:56.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:55 smithi077 ceph-mon[22428]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:56.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:55 smithi077 ceph-mon[27166]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:55:56.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:55 smithi077 ceph-mon[27166]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:55:58.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:57 smithi170 ceph-mon[23272]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:58.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:57 smithi077 ceph-mon[22428]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:55:58.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:57 smithi077 ceph-mon[27166]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:00.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:55:59 smithi077 ceph-mon[22428]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:00.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:55:59 smithi077 ceph-mon[27166]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:00.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:55:59 smithi170 ceph-mon[23272]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:02.103 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:01 smithi077 ceph-mon[22428]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:02.104 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:01 smithi077 ceph-mon[27166]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:02.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:01 smithi170 ceph-mon[23272]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:03.120 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:02 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:56:03.120 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:02 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:56:03.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:02 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:56:03.802 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:56:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:56:03] "GET /metrics HTTP/1.1" 200 37997 "" "Prometheus/2.43.0" 2024-05-06T13:56:04.262 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:03 smithi077 ceph-mon[22428]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:04.263 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:03 smithi077 ceph-mon[27166]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:04.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:03 smithi170 ceph-mon[23272]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:06.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:05 smithi077 ceph-mon[22428]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:56:06.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:05 smithi077 ceph-mon[22428]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:06.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:05 smithi077 ceph-mon[27166]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:56:06.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:05 smithi077 ceph-mon[27166]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:06.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:05 smithi170 ceph-mon[23272]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:56:06.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:05 smithi170 ceph-mon[23272]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:08.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:07 smithi077 ceph-mon[22428]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:08.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:07 smithi077 ceph-mon[27166]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:08.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:07 smithi170 ceph-mon[23272]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:10.129 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:09 smithi170 ceph-mon[23272]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:10.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:09 smithi077 ceph-mon[22428]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:10.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:09 smithi077 ceph-mon[27166]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:11.779 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-05-06T13:56:12.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:11 smithi077 ceph-mon[22428]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:12.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:11 smithi077 ceph-mon[27166]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:12.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:11 smithi170 ceph-mon[23272]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:13.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:12 smithi077 ceph-mon[22428]: from='client.15786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:56:13.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:12 smithi077 ceph-mon[27166]: from='client.15786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:56:13.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:12 smithi170 ceph-mon[23272]: from='client.15786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:56:13.894 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T13:56:14.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:13 smithi077 ceph-mon[22428]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:14.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:13 smithi077 ceph-mon[27166]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:14.053 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:56:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:56:13] "GET /metrics HTTP/1.1" 200 37997 "" "Prometheus/2.43.0" 2024-05-06T13:56:14.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:13 smithi170 ceph-mon[23272]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:14.532 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-05-06T13:56:15.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:14 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/1421199349' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:15.009 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:14 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/1421199349' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:15.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:14 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/1421199349' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:16.185 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:15 smithi077 ceph-mon[22428]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:56:16.185 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:15 smithi077 ceph-mon[22428]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:16.186 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:15 smithi077 ceph-mon[27166]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:56:16.186 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:15 smithi077 ceph-mon[27166]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:16.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:15 smithi170 ceph-mon[23272]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:56:16.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:15 smithi170 ceph-mon[23272]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:16.805 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)" 2024-05-06T13:56:17.150 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:16 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/834163093' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:17.150 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:16 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/834163093' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:17.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:16 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/834163093' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:17.475 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 2'"'"'' 2024-05-06T13:56:18.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:17 smithi077 ceph-mon[27166]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:18.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:17 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:56:18.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:17 smithi077 ceph-mon[22428]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:18.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:17 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:56:18.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:17 smithi170 ceph-mon[23272]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:18.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:17 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:56:19.533 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T13:56:20.141 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:19 smithi077 ceph-mon[22428]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:20.142 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:19 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2643148118' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:20.142 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:19 smithi077 ceph-mon[27166]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:20.142 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:19 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2643148118' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:20.143 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 2'"'"'' 2024-05-06T13:56:20.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:19 smithi170 ceph-mon[23272]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:20.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:19 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2643148118' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:21.979 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:21 smithi077 ceph-mon[22428]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:21.980 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:21 smithi077 ceph-mon[27166]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:22.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:21 smithi170 ceph-mon[23272]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:23.722 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T13:56:24.007 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:23 smithi077 ceph-mon[22428]: from='client.15810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:56:24.007 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:23 smithi077 ceph-mon[22428]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:24.007 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:56:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:56:23] "GET /metrics HTTP/1.1" 200 37993 "" "Prometheus/2.43.0" 2024-05-06T13:56:24.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:23 smithi077 ceph-mon[27166]: from='client.15810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:56:24.008 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:23 smithi077 ceph-mon[27166]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:24.313 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch upgrade status' 2024-05-06T13:56:24.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:23 smithi170 ceph-mon[23272]: from='client.15810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:56:24.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:23 smithi170 ceph-mon[23272]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:26.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:25 smithi077 ceph-mon[22428]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:56:26.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:25 smithi077 ceph-mon[22428]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:26.301 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:25 smithi077 ceph-mon[27166]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:56:26.301 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:25 smithi077 ceph-mon[27166]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:26.301 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:56:26.302 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": null, 2024-05-06T13:56:26.302 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": false, 2024-05-06T13:56:26.302 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "", 2024-05-06T13:56:26.302 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [], 2024-05-06T13:56:26.302 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": null, 2024-05-06T13:56:26.302 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "", 2024-05-06T13:56:26.302 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-05-06T13:56:26.302 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:56:26.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:25 smithi170 ceph-mon[23272]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:56:26.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:25 smithi170 ceph-mon[23272]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:26.916 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph health detail' 2024-05-06T13:56:28.185 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:27 smithi077 ceph-mon[22428]: from='client.25717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:56:28.185 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:27 smithi077 ceph-mon[22428]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:28.185 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:27 smithi077 ceph-mon[27166]: from='client.25717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:56:28.186 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:27 smithi077 ceph-mon[27166]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:28.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:27 smithi170 ceph-mon[23272]: from='client.25717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:56:28.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:27 smithi170 ceph-mon[23272]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:28.976 INFO:teuthology.orchestra.run.smithi077.stdout:HEALTH_OK 2024-05-06T13:56:29.633 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk '"'"'{print $2}'"'"')' 2024-05-06T13:56:30.171 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:29 smithi077 ceph-mon[22428]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:30.171 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:29 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/393528752' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T13:56:30.172 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:29 smithi077 ceph-mon[27166]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:30.172 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:29 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/393528752' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T13:56:30.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:29 smithi170 ceph-mon[23272]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:30.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:29 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/393528752' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T13:56:32.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:31 smithi077 ceph-mon[22428]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:32.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:31 smithi077 ceph-mon[27166]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:32.330 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:31 smithi170 ceph-mon[23272]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:33.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:32 smithi077 ceph-mon[22428]: from='client.15828 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:56:33.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:32 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:56:33.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:32 smithi077 ceph-mon[27166]: from='client.15828 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:56:33.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:32 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:56:33.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:32 smithi170 ceph-mon[23272]: from='client.15828 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:56:33.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:32 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:56:33.742 INFO:teuthology.orchestra.run.smithi077.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:56:34.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:56:33 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:56:33] "GET /metrics HTTP/1.1" 200 37995 "" "Prometheus/2.43.0" 2024-05-06T13:56:34.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:33 smithi077 ceph-mon[27166]: from='client.25735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "daemon_types": "mon", "hosts": "smithi170", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:56:34.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:33 smithi077 ceph-mon[27166]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:34.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:33 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:34.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:33 smithi077 ceph-mon[22428]: from='client.25735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "daemon_types": "mon", "hosts": "smithi170", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:56:34.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:33 smithi077 ceph-mon[22428]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:34.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:33 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:34.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:33 smithi170 ceph-mon[23272]: from='client.25735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "daemon_types": "mon", "hosts": "smithi170", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:56:34.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:33 smithi170 ceph-mon[23272]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:34.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:33 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:34.383 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-05-06T13:56:34.944 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:34 smithi077 ceph-mon[27166]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:56:34.944 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:34 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:56:34.944 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:34 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:56:34.944 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:34 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:34.944 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:34 smithi077 ceph-mon[22428]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:56:34.944 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:34 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:56:34.944 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:34 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:56:34.944 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:34 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:35.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:34 smithi170 ceph-mon[23272]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:56:35.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:34 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:56:35.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:34 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:56:35.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:34 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:36.006 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:35 smithi077 ceph-mon[22428]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:56:36.007 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:35 smithi077 ceph-mon[22428]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:56:36.007 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:35 smithi077 ceph-mon[22428]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:36.007 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:35 smithi077 ceph-mon[27166]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:56:36.007 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:35 smithi077 ceph-mon[27166]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:56:36.007 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:35 smithi077 ceph-mon[27166]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:36.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:35 smithi170 ceph-mon[23272]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:56:36.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:35 smithi170 ceph-mon[23272]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:56:36.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:35 smithi170 ceph-mon[23272]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:37.447 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T13:56:37.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:37 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:37.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:37 smithi170 ceph-mon[23272]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (squid) 2024-05-06T13:56:37.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:37 smithi170 ceph-mon[23272]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T13:56:37.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:37 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:56:37.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:37 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:37.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:37 smithi170 ceph-mon[23272]: Upgrade: Setting container_image for all mgr 2024-05-06T13:56:37.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:37 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:37.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:37 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-05-06T13:56:37.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:37 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-05-06T13:56:37.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:37 smithi170 ceph-mon[23272]: Upgrade: It appears safe to stop mon.b 2024-05-06T13:56:37.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:37 smithi170 ceph-mon[23272]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:37.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:37 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:37.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:37 smithi077 ceph-mon[22428]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (squid) 2024-05-06T13:56:37.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:37 smithi077 ceph-mon[22428]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T13:56:37.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:37 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:56:37.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:37 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:37.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:37 smithi077 ceph-mon[22428]: Upgrade: Setting container_image for all mgr 2024-05-06T13:56:37.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:37 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:37.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:37 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-05-06T13:56:37.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:37 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-05-06T13:56:37.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:37 smithi077 ceph-mon[22428]: Upgrade: It appears safe to stop mon.b 2024-05-06T13:56:37.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:37 smithi077 ceph-mon[22428]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:37.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:37 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:37.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:37 smithi077 ceph-mon[27166]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (squid) 2024-05-06T13:56:37.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:37 smithi077 ceph-mon[27166]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T13:56:37.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:37 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:56:37.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:37 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:37.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:37 smithi077 ceph-mon[27166]: Upgrade: Setting container_image for all mgr 2024-05-06T13:56:37.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:37 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:37.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:37 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-05-06T13:56:37.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:37 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-05-06T13:56:37.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:37 smithi077 ceph-mon[27166]: Upgrade: It appears safe to stop mon.b 2024-05-06T13:56:37.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:37 smithi077 ceph-mon[27166]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:38.093 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T13:56:38.094 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (85s) 80s ago 23m 17.1M - 0.25.0 c8568f914cd2 997175e3e64e 2024-05-06T13:56:38.094 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (63s) 57s ago 23m 53.6M - 9.4.12 2bacad6d85d8 140bb9a2760a 2024-05-06T13:56:38.094 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (2m) 80s ago 22m 52.4M - 3.5 e1d6a67b021e 9353a303fb55 2024-05-06T13:56:38.094 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283,8765 running (2m) 57s ago 28m 452M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 125521be53a3 2024-05-06T13:56:38.094 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (12m) 80s ago 29m 504M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T13:56:38.094 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (29m) 80s ago 30m 78.0M 2048M 17.2.0 e1d6a67b021e ac46f91694f8 2024-05-06T13:56:38.094 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (28m) 57s ago 28m 56.1M 2048M 17.2.0 e1d6a67b021e 24be6fce9d28 2024-05-06T13:56:38.094 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (28m) 80s ago 28m 57.2M 2048M 17.2.0 e1d6a67b021e 83af336ef762 2024-05-06T13:56:38.094 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (2m) 80s ago 24m 14.4M - 1.5.0 0da6a335fe13 67f45fa4968b 2024-05-06T13:56:38.094 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (2m) 57s ago 24m 16.0M - 1.5.0 0da6a335fe13 02081b71a5dd 2024-05-06T13:56:38.094 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (28m) 80s ago 27m 54.4M 2992M 17.2.0 e1d6a67b021e 2ccee2bb6e96 2024-05-06T13:56:38.094 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (27m) 80s ago 27m 58.5M 2992M 17.2.0 e1d6a67b021e 1d9e6609b585 2024-05-06T13:56:38.094 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (27m) 80s ago 27m 52.4M 2992M 17.2.0 e1d6a67b021e 37a61d295111 2024-05-06T13:56:38.095 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (26m) 80s ago 26m 52.9M 2992M 17.2.0 e1d6a67b021e 8feb7c077aaa 2024-05-06T13:56:38.095 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (26m) 57s ago 26m 58.1M 3248M 17.2.0 e1d6a67b021e f865acc44a06 2024-05-06T13:56:38.095 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (26m) 57s ago 26m 52.5M 3248M 17.2.0 e1d6a67b021e a55bd4122f6f 2024-05-06T13:56:38.095 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (25m) 57s ago 25m 51.2M 3248M 17.2.0 e1d6a67b021e 362938a0b357 2024-05-06T13:56:38.095 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (25m) 57s ago 25m 53.7M 3248M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T13:56:38.095 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (108s) 57s ago 24m 51.5M - 2.43.0 a07b618ecd1d 10d5c513bb9b 2024-05-06T13:56:38.095 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (23m) 80s ago 23m 77.3M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T13:56:38.095 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (22m) 57s ago 22m 76.3M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T13:56:38.491 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:56:38.491 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-05-06T13:56:38.491 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-05-06T13:56:38.491 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:56:38.491 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-05-06T13:56:38.492 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T13:56:38.492 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:56:38.492 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-05-06T13:56:38.492 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-06T13:56:38.492 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:56:38.492 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": {}, 2024-05-06T13:56:38.492 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw": { 2024-05-06T13:56:38.492 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T13:56:38.492 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:56:38.492 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-05-06T13:56:38.492 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-05-06T13:56:38.492 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T13:56:38.492 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T13:56:38.492 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:56:38.670 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:38 smithi170 ceph-mon[23272]: from='client.15840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:56:38.670 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:38 smithi170 ceph-mon[23272]: from='client.15846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:56:38.670 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:38 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:38.670 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:38 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:56:38.670 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:38 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T13:56:38.670 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:38 smithi170 ceph-mon[23272]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:56:38.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:38 smithi077 ceph-mon[22428]: from='client.15840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:56:38.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:38 smithi077 ceph-mon[22428]: from='client.15846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:56:38.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:38 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:38.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:38 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:56:38.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:38 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T13:56:38.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:38 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:56:38.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:38 smithi077 ceph-mon[27166]: from='client.15840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:56:38.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:38 smithi077 ceph-mon[27166]: from='client.15846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:56:38.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:38 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:38.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:38 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:56:38.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:38 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T13:56:38.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:38 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:56:38.822 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:56:38.822 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T13:56:38.822 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-05-06T13:56:38.822 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi170", 2024-05-06T13:56:38.822 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [], 2024-05-06T13:56:38.822 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "0/1 daemons upgraded", 2024-05-06T13:56:38.822 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "Currently upgrading mon daemons", 2024-05-06T13:56:38.822 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-05-06T13:56:38.822 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:56:39.699 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:39 smithi170 ceph-mon[23272]: from='client.15852 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:56:39.699 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:39 smithi170 ceph-mon[23272]: Upgrade: Updating mon.b 2024-05-06T13:56:39.699 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:39 smithi170 ceph-mon[23272]: Deploying daemon mon.b on smithi170 2024-05-06T13:56:39.699 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:39 smithi170 ceph-mon[23272]: from='client.? 172.21.15.77:0/2268045034' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:39.699 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:39 smithi170 ceph-mon[23272]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:39.699 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:39 smithi170 ceph-mon[23272]: from='client.25765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:56:39.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:39 smithi077 ceph-mon[22428]: from='client.15852 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:56:39.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:39 smithi077 ceph-mon[22428]: Upgrade: Updating mon.b 2024-05-06T13:56:39.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:39 smithi077 ceph-mon[22428]: Deploying daemon mon.b on smithi170 2024-05-06T13:56:39.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:39 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/2268045034' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:39.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:39 smithi077 ceph-mon[22428]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:39.804 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:39 smithi077 ceph-mon[22428]: from='client.25765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:56:39.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:39 smithi077 ceph-mon[27166]: from='client.15852 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:56:39.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:39 smithi077 ceph-mon[27166]: Upgrade: Updating mon.b 2024-05-06T13:56:39.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:39 smithi077 ceph-mon[27166]: Deploying daemon mon.b on smithi170 2024-05-06T13:56:39.805 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:39 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/2268045034' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:39.805 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:39 smithi077 ceph-mon[27166]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:39.805 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:39 smithi077 ceph-mon[27166]: from='client.25765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:56:40.238 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:39 smithi170 systemd[1]: Stopping Ceph mon.b for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:56:40.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:40 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-b[23268]: 2024-05-06T13:56:40.236+0000 7f5805c99700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T13:56:40.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:40 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-b[23268]: 2024-05-06T13:56:40.236+0000 7f5805c99700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-05-06T13:56:40.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:40 smithi170 podman[65695]: 2024-05-06 13:56:40.348906803 +0000 UTC m=+0.231975285 container died 24be6fce9d28a1fec954db1eb8cf7b5583295d6e779e9ae383c08f400988d439 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-b, build-date=2022-05-03T08:36:31.336870, release=754, RELEASE=HEAD, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, name=centos-stream, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True) 2024-05-06T13:56:41.030 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:40 smithi170 podman[65695]: 2024-05-06 13:56:40.71158366 +0000 UTC m=+0.594652150 container cleanup 24be6fce9d28a1fec954db1eb8cf7b5583295d6e779e9ae383c08f400988d439 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-b, RELEASE=HEAD, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, build-date=2022-05-03T08:36:31.336870, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, architecture=x86_64, GIT_BRANCH=HEAD, io.openshift.expose-services=, io.buildah.version=1.19.8, release=754, io.k8s.display-name=CentOS Stream 8, vcs-type=git, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container) 2024-05-06T13:56:41.030 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:40 smithi170 bash[65695]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-b 2024-05-06T13:56:41.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 podman[65709]: 2024-05-06 13:56:41.030080215 +0000 UTC m=+0.679695565 container remove 24be6fce9d28a1fec954db1eb8cf7b5583295d6e779e9ae383c08f400988d439 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-b, io.openshift.expose-services=, version=8, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., architecture=x86_64, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, name=centos-stream, distribution-scope=public, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870) 2024-05-06T13:56:41.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mon.b.service: Deactivated successfully. 2024-05-06T13:56:41.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 systemd[1]: Stopped Ceph mon.b for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:56:41.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mon.b.service: Consumed 17.772s CPU time. 2024-05-06T13:56:41.739 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 systemd[1]: Starting Ceph mon.b for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:56:41.739 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 podman[65837]: 2024-05-06 13:56:41.518451914 +0000 UTC m=+0.018951062 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:56:41.739 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 podman[65837]: 2024-05-06 13:56:41.615064848 +0000 UTC m=+0.115563965 container create 61cd48b3d0ff22847979c7591e6adaec1ad30037835ff47d257f3146ffc3d053 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-b, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T13:56:42.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 podman[65837]: 2024-05-06 13:56:41.851220887 +0000 UTC m=+0.351720005 container init 61cd48b3d0ff22847979c7591e6adaec1ad30037835ff47d257f3146ffc3d053 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-06T13:56:42.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 podman[65837]: 2024-05-06 13:56:41.85469309 +0000 UTC m=+0.355192214 container start 61cd48b3d0ff22847979c7591e6adaec1ad30037835ff47d257f3146ffc3d053 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-b, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-05-06T13:56:42.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: set uid:gid to 167:167 (ceph:ceph) 2024-05-06T13:56:42.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev), process ceph-mon, pid 2 2024-05-06T13:56:42.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: pidfile_write: ignore empty --pid-file 2024-05-06T13:56:42.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: load: jerasure load: lrc 2024-05-06T13:56:42.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: RocksDB version: 7.9.2 2024-05-06T13:56:42.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Git sha 0 2024-05-06T13:56:42.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Compile date 2024-05-03 15:08:47 2024-05-06T13:56:42.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: DB SUMMARY 2024-05-06T13:56:42.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: DB Session ID: J2NF2OR9747Y9H0QNTSQ 2024-05-06T13:56:42.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: CURRENT file: CURRENT 2024-05-06T13:56:42.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: IDENTITY file: IDENTITY 2024-05-06T13:56:42.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: MANIFEST file: MANIFEST-000009 size: 2822 Bytes 2024-05-06T13:56:42.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000054.sst 2024-05-06T13:56:42.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000052.log size: 2534679 ; 2024-05-06T13:56:42.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.error_if_exists: 0 2024-05-06T13:56:42.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.create_if_missing: 0 2024-05-06T13:56:42.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.paranoid_checks: 1 2024-05-06T13:56:42.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-06T13:56:42.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-06T13:56:42.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-06T13:56:42.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.env: 0x558dc855fc60 2024-05-06T13:56:42.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.fs: PosixFileSystem 2024-05-06T13:56:42.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.info_log: 0x558dca953ac0 2024-05-06T13:56:42.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.max_file_opening_threads: 16 2024-05-06T13:56:42.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.statistics: (nil) 2024-05-06T13:56:42.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.use_fsync: 0 2024-05-06T13:56:42.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.max_log_file_size: 0 2024-05-06T13:56:42.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-06T13:56:42.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-06T13:56:42.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.keep_log_file_num: 1000 2024-05-06T13:56:42.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.recycle_log_file_num: 0 2024-05-06T13:56:42.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.allow_fallocate: 1 2024-05-06T13:56:42.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.allow_mmap_reads: 0 2024-05-06T13:56:42.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.allow_mmap_writes: 0 2024-05-06T13:56:42.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.use_direct_reads: 0 2024-05-06T13:56:42.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-06T13:56:42.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.create_missing_column_families: 0 2024-05-06T13:56:42.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.db_log_dir: 2024-05-06T13:56:42.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.wal_dir: 2024-05-06T13:56:42.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-06T13:56:42.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-06T13:56:42.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-06T13:56:42.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-06T13:56:42.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-06T13:56:42.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-06T13:56:42.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.advise_random_on_open: 1 2024-05-06T13:56:42.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.db_write_buffer_size: 0 2024-05-06T13:56:42.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.write_buffer_manager: 0x558dca926820 2024-05-06T13:56:42.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-06T13:56:42.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-06T13:56:42.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-06T13:56:42.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.rate_limiter: (nil) 2024-05-06T13:56:42.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-06T13:56:42.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.wal_recovery_mode: 2 2024-05-06T13:56:42.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.enable_thread_tracking: 0 2024-05-06T13:56:42.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.enable_pipelined_write: 0 2024-05-06T13:56:42.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.unordered_write: 0 2024-05-06T13:56:42.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-06T13:56:42.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-06T13:56:42.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-06T13:56:42.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-06T13:56:42.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.row_cache: None 2024-05-06T13:56:42.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.wal_filter: None 2024-05-06T13:56:42.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-06T13:56:42.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.allow_ingest_behind: 0 2024-05-06T13:56:42.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.two_write_queues: 0 2024-05-06T13:56:42.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.manual_wal_flush: 0 2024-05-06T13:56:42.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.wal_compression: 0 2024-05-06T13:56:42.083 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.atomic_flush: 0 2024-05-06T13:56:42.083 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-06T13:56:42.083 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-06T13:56:42.083 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-06T13:56:42.083 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.log_readahead_size: 0 2024-05-06T13:56:42.083 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-06T13:56:42.083 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.best_efforts_recovery: 0 2024-05-06T13:56:42.083 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-06T13:56:42.083 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-06T13:56:42.083 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.allow_data_in_errors: 0 2024-05-06T13:56:42.083 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.db_host_id: __hostname__ 2024-05-06T13:56:42.083 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-06T13:56:42.083 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.max_background_jobs: 2 2024-05-06T13:56:42.083 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.max_background_compactions: -1 2024-05-06T13:56:42.083 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.max_subcompactions: 1 2024-05-06T13:56:42.084 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-06T13:56:42.084 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-06T13:56:42.084 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-06T13:56:42.084 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.max_total_wal_size: 0 2024-05-06T13:56:42.084 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-06T13:56:42.084 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-06T13:56:42.084 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-06T13:56:42.084 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-06T13:56:42.084 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.max_open_files: -1 2024-05-06T13:56:42.084 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.bytes_per_sync: 0 2024-05-06T13:56:42.084 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-06T13:56:42.084 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-06T13:56:42.084 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.compaction_readahead_size: 0 2024-05-06T13:56:42.084 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.max_background_flushes: -1 2024-05-06T13:56:42.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Compression algorithms supported: 2024-05-06T13:56:42.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-06T13:56:42.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: kZSTD supported: 0 2024-05-06T13:56:42.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: kXpressCompression supported: 0 2024-05-06T13:56:42.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: kLZ4HCCompression supported: 1 2024-05-06T13:56:42.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: kZlibCompression supported: 1 2024-05-06T13:56:42.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: kSnappyCompression supported: 1 2024-05-06T13:56:42.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: kLZ4Compression supported: 1 2024-05-06T13:56:42.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: kBZip2Compression supported: 0 2024-05-06T13:56:42.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-06T13:56:42.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-06T13:56:42.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-05-06T13:56:42.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-06T13:56:42.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-06T13:56:42.085 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.merge_operator: 2024-05-06T13:56:42.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.compaction_filter: None 2024-05-06T13:56:42.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.compaction_filter_factory: None 2024-05-06T13:56:42.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.sst_partitioner_factory: None 2024-05-06T13:56:42.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-06T13:56:42.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-06T13:56:42.086 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558dca953bc0) 2024-05-06T13:56:42.086 INFO:journalctl@ceph.mon.b.smithi170.stdout: cache_index_and_filter_blocks: 1 2024-05-06T13:56:42.086 INFO:journalctl@ceph.mon.b.smithi170.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-06T13:56:42.086 INFO:journalctl@ceph.mon.b.smithi170.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-06T13:56:42.086 INFO:journalctl@ceph.mon.b.smithi170.stdout: pin_top_level_index_and_filter: 1 2024-05-06T13:56:42.086 INFO:journalctl@ceph.mon.b.smithi170.stdout: index_type: 0 2024-05-06T13:56:42.086 INFO:journalctl@ceph.mon.b.smithi170.stdout: data_block_index_type: 0 2024-05-06T13:56:42.086 INFO:journalctl@ceph.mon.b.smithi170.stdout: index_shortening: 1 2024-05-06T13:56:42.086 INFO:journalctl@ceph.mon.b.smithi170.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-06T13:56:42.086 INFO:journalctl@ceph.mon.b.smithi170.stdout: checksum: 4 2024-05-06T13:56:42.087 INFO:journalctl@ceph.mon.b.smithi170.stdout: no_block_cache: 0 2024-05-06T13:56:42.087 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_cache: 0x558dca957090 2024-05-06T13:56:42.087 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_cache_name: BinnedLRUCache 2024-05-06T13:56:42.087 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_cache_options: 2024-05-06T13:56:42.087 INFO:journalctl@ceph.mon.b.smithi170.stdout: capacity : 536870912 2024-05-06T13:56:42.087 INFO:journalctl@ceph.mon.b.smithi170.stdout: num_shard_bits : 4 2024-05-06T13:56:42.087 INFO:journalctl@ceph.mon.b.smithi170.stdout: strict_capacity_limit : 0 2024-05-06T13:56:42.087 INFO:journalctl@ceph.mon.b.smithi170.stdout: high_pri_pool_ratio: 0.000 2024-05-06T13:56:42.087 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_cache_compressed: (nil) 2024-05-06T13:56:42.087 INFO:journalctl@ceph.mon.b.smithi170.stdout: persistent_cache: (nil) 2024-05-06T13:56:42.087 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_size: 4096 2024-05-06T13:56:42.087 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_size_deviation: 10 2024-05-06T13:56:42.087 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_restart_interval: 16 2024-05-06T13:56:42.087 INFO:journalctl@ceph.mon.b.smithi170.stdout: index_block_restart_interval: 1 2024-05-06T13:56:42.087 INFO:journalctl@ceph.mon.b.smithi170.stdout: metadata_block_size: 4096 2024-05-06T13:56:42.088 INFO:journalctl@ceph.mon.b.smithi170.stdout: partition_filters: 0 2024-05-06T13:56:42.088 INFO:journalctl@ceph.mon.b.smithi170.stdout: use_delta_encoding: 1 2024-05-06T13:56:42.088 INFO:journalctl@ceph.mon.b.smithi170.stdout: filter_policy: bloomfilter 2024-05-06T13:56:42.088 INFO:journalctl@ceph.mon.b.smithi170.stdout: whole_key_filtering: 1 2024-05-06T13:56:42.088 INFO:journalctl@ceph.mon.b.smithi170.stdout: verify_compression: 0 2024-05-06T13:56:42.088 INFO:journalctl@ceph.mon.b.smithi170.stdout: read_amp_bytes_per_bit: 0 2024-05-06T13:56:42.088 INFO:journalctl@ceph.mon.b.smithi170.stdout: format_version: 5 2024-05-06T13:56:42.088 INFO:journalctl@ceph.mon.b.smithi170.stdout: enable_index_compression: 1 2024-05-06T13:56:42.088 INFO:journalctl@ceph.mon.b.smithi170.stdout: block_align: 0 2024-05-06T13:56:42.088 INFO:journalctl@ceph.mon.b.smithi170.stdout: max_auto_readahead_size: 262144 2024-05-06T13:56:42.088 INFO:journalctl@ceph.mon.b.smithi170.stdout: prepopulate_block_cache: 0 2024-05-06T13:56:42.088 INFO:journalctl@ceph.mon.b.smithi170.stdout: initial_auto_readahead_size: 8192 2024-05-06T13:56:42.088 INFO:journalctl@ceph.mon.b.smithi170.stdout: num_file_reads_for_auto_readahead: 2 2024-05-06T13:56:42.088 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.write_buffer_size: 33554432 2024-05-06T13:56:42.088 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.max_write_buffer_number: 2 2024-05-06T13:56:42.088 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.compression: NoCompression 2024-05-06T13:56:42.089 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.bottommost_compression: Disabled 2024-05-06T13:56:42.089 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.prefix_extractor: nullptr 2024-05-06T13:56:42.089 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-06T13:56:42.089 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.num_levels: 7 2024-05-06T13:56:42.089 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-06T13:56:42.089 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-06T13:56:42.089 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-06T13:56:42.089 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-06T13:56:42.089 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-06T13:56:42.089 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-06T13:56:42.089 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-06T13:56:42.089 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-06T13:56:42.089 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-06T13:56:42.089 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-06T13:56:42.089 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-06T13:56:42.090 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-06T13:56:42.090 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-06T13:56:42.090 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.compression_opts.level: 32767 2024-05-06T13:56:42.090 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.compression_opts.strategy: 0 2024-05-06T13:56:42.090 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-06T13:56:42.090 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-06T13:56:42.090 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-06T13:56:42.090 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-06T13:56:42.090 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.compression_opts.enabled: false 2024-05-06T13:56:42.090 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-06T13:56:42.090 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-06T13:56:42.090 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-06T13:56:42.090 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-06T13:56:42.090 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.target_file_size_base: 67108864 2024-05-06T13:56:42.091 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-06T13:56:42.091 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-06T13:56:42.091 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-06T13:56:42.091 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-06T13:56:42.091 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-06T13:56:42.091 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-06T13:56:42.091 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-06T13:56:42.092 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-06T13:56:42.092 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-06T13:56:42.092 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-06T13:56:42.092 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-06T13:56:42.092 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-06T13:56:42.092 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-06T13:56:42.092 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-06T13:56:42.092 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.arena_block_size: 1048576 2024-05-06T13:56:42.092 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-06T13:56:42.092 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-06T13:56:42.092 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.disable_auto_compactions: 0 2024-05-06T13:56:42.093 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-06T13:56:42.093 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-06T13:56:42.093 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-06T13:56:42.093 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-06T13:56:42.093 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-06T13:56:42.093 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-06T13:56:42.093 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-06T13:56:42.093 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-06T13:56:42.093 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-06T13:56:42.093 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-06T13:56:42.093 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.table_properties_collectors: 2024-05-06T13:56:42.093 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.inplace_update_support: 0 2024-05-06T13:56:42.093 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-06T13:56:42.093 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-06T13:56:42.094 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-06T13:56:42.094 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-06T13:56:42.094 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.bloom_locality: 0 2024-05-06T13:56:42.094 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.max_successive_merges: 0 2024-05-06T13:56:42.094 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-06T13:56:42.094 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.paranoid_file_checks: 0 2024-05-06T13:56:42.094 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.force_consistency_checks: 1 2024-05-06T13:56:42.094 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.report_bg_io_stats: 0 2024-05-06T13:56:42.094 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.ttl: 2592000 2024-05-06T13:56:42.094 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-06T13:56:42.094 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-06T13:56:42.094 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-06T13:56:42.094 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.enable_blob_files: false 2024-05-06T13:56:42.094 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.min_blob_size: 0 2024-05-06T13:56:42.094 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.blob_file_size: 268435456 2024-05-06T13:56:42.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-06T13:56:42.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-06T13:56:42.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-06T13:56:42.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-06T13:56:42.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-06T13:56:42.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.blob_file_starting_level: 0 2024-05-06T13:56:42.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-06T13:56:42.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 54.sst 2024-05-06T13:56:42.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-05-06T13:56:42.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 56, last_sequence is 28936, log_number is 52,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-06T13:56:42.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 52 2024-05-06T13:56:42.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7f08b7b2-1da9-4c9e-ad42-a93150ab2feb 2024-05-06T13:56:42.095 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715003801884067, "job": 1, "event": "recovery_started", "wal_files": [52]} 2024-05-06T13:56:42.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #52 mode 2 2024-05-06T13:56:42.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715003801890933, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 57, "file_size": 1573519, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 28941, "largest_seqno": 30541, "table_properties": {"data_size": 1567477, "index_size": 3491, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1541, "raw_key_size": 14455, "raw_average_key_size": 24, "raw_value_size": 1554883, "raw_average_value_size": 2622, "num_data_blocks": 158, "num_entries": 593, "num_filter_entries": 593, "num_deletions": 7, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715003801, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7f08b7b2-1da9-4c9e-ad42-a93150ab2feb", "db_session_id": "J2NF2OR9747Y9H0QNTSQ", "orig_file_number": 57, "seqno_to_time_mapping": "N/A"}} 2024-05-06T13:56:42.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715003801891037, "job": 1, "event": "recovery_finished"} 2024-05-06T13:56:42.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: [db/version_set.cc:5047] Creating manifest 59 2024-05-06T13:56:42.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-05-06T13:56:42.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000052.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-06T13:56:42.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558dcaa48000 2024-05-06T13:56:42.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: DB pointer 0x558dcaa32000 2024-05-06T13:56:42.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-06T13:56:42.096 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-06T13:56:42.096 INFO:journalctl@ceph.mon.b.smithi170.stdout: ** DB Stats ** 2024-05-06T13:56:42.096 INFO:journalctl@ceph.mon.b.smithi170.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T13:56:42.096 INFO:journalctl@ceph.mon.b.smithi170.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-06T13:56:42.096 INFO:journalctl@ceph.mon.b.smithi170.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-06T13:56:42.097 INFO:journalctl@ceph.mon.b.smithi170.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-06T13:56:42.097 INFO:journalctl@ceph.mon.b.smithi170.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-06T13:56:42.097 INFO:journalctl@ceph.mon.b.smithi170.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-06T13:56:42.097 INFO:journalctl@ceph.mon.b.smithi170.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-06T13:56:42.097 INFO:journalctl@ceph.mon.b.smithi170.stdout: 2024-05-06T13:56:42.097 INFO:journalctl@ceph.mon.b.smithi170.stdout: ** Compaction Stats [default] ** 2024-05-06T13:56:42.097 INFO:journalctl@ceph.mon.b.smithi170.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-06T13:56:42.097 INFO:journalctl@ceph.mon.b.smithi170.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-06T13:56:42.097 INFO:journalctl@ceph.mon.b.smithi170.stdout: L0 1/0 1.50 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 469.5 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-05-06T13:56:42.097 INFO:journalctl@ceph.mon.b.smithi170.stdout: L6 1/0 8.89 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-05-06T13:56:42.097 INFO:journalctl@ceph.mon.b.smithi170.stdout: Sum 2/0 10.39 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 469.5 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-05-06T13:56:42.097 INFO:journalctl@ceph.mon.b.smithi170.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 469.5 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-05-06T13:56:42.097 INFO:journalctl@ceph.mon.b.smithi170.stdout: 2024-05-06T13:56:42.097 INFO:journalctl@ceph.mon.b.smithi170.stdout: ** Compaction Stats [default] ** 2024-05-06T13:56:42.097 INFO:journalctl@ceph.mon.b.smithi170.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-06T13:56:42.098 INFO:journalctl@ceph.mon.b.smithi170.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-06T13:56:42.098 INFO:journalctl@ceph.mon.b.smithi170.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 469.5 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-05-06T13:56:42.098 INFO:journalctl@ceph.mon.b.smithi170.stdout: 2024-05-06T13:56:42.098 INFO:journalctl@ceph.mon.b.smithi170.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-06T13:56:42.098 INFO:journalctl@ceph.mon.b.smithi170.stdout: 2024-05-06T13:56:42.098 INFO:journalctl@ceph.mon.b.smithi170.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T13:56:42.098 INFO:journalctl@ceph.mon.b.smithi170.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-05-06T13:56:42.098 INFO:journalctl@ceph.mon.b.smithi170.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-06T13:56:42.098 INFO:journalctl@ceph.mon.b.smithi170.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-06T13:56:42.098 INFO:journalctl@ceph.mon.b.smithi170.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-06T13:56:42.098 INFO:journalctl@ceph.mon.b.smithi170.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-06T13:56:42.098 INFO:journalctl@ceph.mon.b.smithi170.stdout: Cumulative compaction: 0.00 GB write, 163.09 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T13:56:42.098 INFO:journalctl@ceph.mon.b.smithi170.stdout: Interval compaction: 0.00 GB write, 163.09 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T13:56:42.098 INFO:journalctl@ceph.mon.b.smithi170.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-06T13:56:42.099 INFO:journalctl@ceph.mon.b.smithi170.stdout: Block cache BinnedLRUCache@0x558dca957090#2 capacity: 512.00 MB usage: 5.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-05-06T13:56:42.099 INFO:journalctl@ceph.mon.b.smithi170.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.78 KB,0.000339746%) IndexBlock(1,4.08 KB,0.000777841%) Misc(1,0.00 KB,0%) 2024-05-06T13:56:42.099 INFO:journalctl@ceph.mon.b.smithi170.stdout: 2024-05-06T13:56:42.099 INFO:journalctl@ceph.mon.b.smithi170.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-06T13:56:42.099 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: starting mon.b rank 2 at public addrs [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] at bind addrs [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:56:42.099 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: mon.b@-1(???) e3 preinit fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:56:42.099 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: mon.b@-1(???).mds e1 new map 2024-05-06T13:56:42.099 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: mon.b@-1(???).mds e1 print_map 2024-05-06T13:56:42.099 INFO:journalctl@ceph.mon.b.smithi170.stdout: e1 2024-05-06T13:56:42.099 INFO:journalctl@ceph.mon.b.smithi170.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-06T13:56:42.099 INFO:journalctl@ceph.mon.b.smithi170.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-06T13:56:42.099 INFO:journalctl@ceph.mon.b.smithi170.stdout: legacy client fscid: -1 2024-05-06T13:56:42.099 INFO:journalctl@ceph.mon.b.smithi170.stdout: 2024-05-06T13:56:42.099 INFO:journalctl@ceph.mon.b.smithi170.stdout: No filesystems configured 2024-05-06T13:56:42.099 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: mon.b@-1(???).osd e112 crush map has features 3314933000854323200, adjusting msgr requires 2024-05-06T13:56:42.100 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: mon.b@-1(???).osd e112 crush map has features 432629239337189376, adjusting msgr requires 2024-05-06T13:56:42.100 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: mon.b@-1(???).osd e112 crush map has features 432629239337189376, adjusting msgr requires 2024-05-06T13:56:42.100 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: mon.b@-1(???).osd e112 crush map has features 432629239337189376, adjusting msgr requires 2024-05-06T13:56:42.100 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 ceph-mon[65866]: mon.b@-1(???).paxosservice(auth 1..26) refresh upgraded, format 0 -> 3 2024-05-06T13:56:42.100 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 bash[65837]: 61cd48b3d0ff22847979c7591e6adaec1ad30037835ff47d257f3146ffc3d053 2024-05-06T13:56:42.100 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:41 smithi170 systemd[1]: Started Ceph mon.b for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:56:43.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:43 smithi077 ceph-mon[22428]: mon.b calling monitor election 2024-05-06T13:56:43.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:43 smithi077 ceph-mon[22428]: mon.c calling monitor election 2024-05-06T13:56:43.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:43 smithi077 ceph-mon[22428]: mon.a calling monitor election 2024-05-06T13:56:43.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:43 smithi077 ceph-mon[22428]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-06T13:56:43.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:43 smithi077 ceph-mon[22428]: monmap e3: 3 mons at {a=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0],b=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0],c=[v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0]} 2024-05-06T13:56:43.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:43 smithi077 ceph-mon[22428]: fsmap 2024-05-06T13:56:43.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:43 smithi077 ceph-mon[22428]: osdmap e112: 8 total, 8 up, 8 in 2024-05-06T13:56:43.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:43 smithi077 ceph-mon[22428]: mgrmap e42: y(active, since 3m), standbys: x 2024-05-06T13:56:43.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:43 smithi077 ceph-mon[22428]: overall HEALTH_OK 2024-05-06T13:56:43.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:43 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:43.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:43 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:43.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:43 smithi077 ceph-mon[22428]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:43.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:43 smithi077 ceph-mon[27166]: mon.b calling monitor election 2024-05-06T13:56:43.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:43 smithi077 ceph-mon[27166]: mon.c calling monitor election 2024-05-06T13:56:43.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:43 smithi077 ceph-mon[27166]: mon.a calling monitor election 2024-05-06T13:56:43.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:43 smithi077 ceph-mon[27166]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-06T13:56:43.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:43 smithi077 ceph-mon[27166]: monmap e3: 3 mons at {a=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0],b=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0],c=[v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0]} 2024-05-06T13:56:43.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:43 smithi077 ceph-mon[27166]: fsmap 2024-05-06T13:56:43.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:43 smithi077 ceph-mon[27166]: osdmap e112: 8 total, 8 up, 8 in 2024-05-06T13:56:43.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:43 smithi077 ceph-mon[27166]: mgrmap e42: y(active, since 3m), standbys: x 2024-05-06T13:56:43.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:43 smithi077 ceph-mon[27166]: overall HEALTH_OK 2024-05-06T13:56:43.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:43 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:43.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:43 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:43.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:43 smithi077 ceph-mon[27166]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:43.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:43 smithi170 ceph-mon[65866]: mon.b calling monitor election 2024-05-06T13:56:43.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:43 smithi170 ceph-mon[65866]: mon.c calling monitor election 2024-05-06T13:56:43.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:43 smithi170 ceph-mon[65866]: mon.a calling monitor election 2024-05-06T13:56:43.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:43 smithi170 ceph-mon[65866]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-06T13:56:43.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:43 smithi170 ceph-mon[65866]: monmap e3: 3 mons at {a=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0],b=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0],c=[v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0]} 2024-05-06T13:56:43.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:43 smithi170 ceph-mon[65866]: fsmap 2024-05-06T13:56:43.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:43 smithi170 ceph-mon[65866]: osdmap e112: 8 total, 8 up, 8 in 2024-05-06T13:56:43.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:43 smithi170 ceph-mon[65866]: mgrmap e42: y(active, since 3m), standbys: x 2024-05-06T13:56:43.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:43 smithi170 ceph-mon[65866]: overall HEALTH_OK 2024-05-06T13:56:43.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:43 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:43.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:43 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:43.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:43 smithi170 ceph-mon[65866]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:44.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:56:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:56:43] "GET /metrics HTTP/1.1" 200 37995 "" "Prometheus/2.43.0" 2024-05-06T13:56:46.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:45 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:56:46.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:45 smithi170 ceph-mon[65866]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:46.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:45 smithi077 ceph-mon[22428]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:56:46.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:45 smithi077 ceph-mon[22428]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:46.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:45 smithi077 ceph-mon[27166]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:56:46.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:45 smithi077 ceph-mon[27166]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:47.302 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:56:46 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:56:46.900+0000 7fb092c58700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-05-06T13:56:48.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:47 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:48.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:47 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:48.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:47 smithi077 ceph-mon[22428]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:48.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:47 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:48.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:47 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:48.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:47 smithi077 ceph-mon[27166]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:48.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:47 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:48.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:47 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:48.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:47 smithi170 ceph-mon[65866]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:48.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:48 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:48.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:48 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:48.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:48 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:56:49.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:48 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:49.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:48 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:49.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:48 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:56:49.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:48 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:49.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:48 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:49.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:48 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:56:50.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:49 smithi077 ceph-mon[22428]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:50.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:49 smithi077 ceph-mon[27166]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:50.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:49 smithi170 ceph-mon[65866]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:52.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:52.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: Detected new or changed devices on smithi170 2024-05-06T13:56:52.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:56:52.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:56:52.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:56:52.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: Upgrade: Setting container_image for all crash 2024-05-06T13:56:52.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: Upgrade: Setting container_image for all mds 2024-05-06T13:56:52.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: Upgrade: Setting container_image for all rbd-mirror 2024-05-06T13:56:52.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-05-06T13:56:52.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-05-06T13:56:52.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-05-06T13:56:52.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:52.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: Detected new or changed devices on smithi170 2024-05-06T13:56:52.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:56:52.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:56:52.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:56:52.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: Upgrade: Setting container_image for all crash 2024-05-06T13:56:52.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: Upgrade: Setting container_image for all mds 2024-05-06T13:56:52.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: Upgrade: Setting container_image for all rbd-mirror 2024-05-06T13:56:52.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-05-06T13:56:52.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-05-06T13:56:52.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-05-06T13:56:52.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-05-06T13:56:52.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-05-06T13:56:52.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-05-06T13:56:52.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-05-06T13:56:52.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-05-06T13:56:52.058 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-05-06T13:56:52.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-05-06T13:56:52.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:56:52.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-05-06T13:56:52.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-05-06T13:56:52.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-05-06T13:56:52.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-05-06T13:56:52.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-05-06T13:56:52.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-05-06T13:56:52.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-05-06T13:56:52.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-05-06T13:56:52.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-05-06T13:56:52.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:56:52.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-05-06T13:56:52.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-05-06T13:56:52.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-05-06T13:56:52.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-05-06T13:56:52.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-05-06T13:56:52.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-05-06T13:56:52.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-05-06T13:56:52.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-05-06T13:56:52.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:56:52.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-05-06T13:56:52.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-05-06T13:56:52.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-05-06T13:56:52.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-05-06T13:56:52.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-05-06T13:56:52.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-05-06T13:56:52.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-05-06T13:56:52.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-05-06T13:56:52.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-05-06T13:56:52.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:56:52.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-05-06T13:56:52.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-05-06T13:56:52.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-05-06T13:56:52.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-05-06T13:56:52.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-05-06T13:56:52.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-05-06T13:56:52.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-05-06T13:56:52.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:56:52.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:56:52.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:56:52.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:56:52.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:56:52.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:56:52.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-05-06T13:56:52.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-05-06T13:56:52.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:56:52.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:56:52.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.063 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:56:52.063 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:56:52.063 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.063 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-05-06T13:56:52.063 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-05-06T13:56:52.063 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-05-06T13:56:52.063 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-05-06T13:56:52.063 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-05-06T13:56:52.063 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-05-06T13:56:52.063 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:56:52.063 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:56:52.063 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:56:52.063 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:56:52.064 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:56:52.064 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:56:52.064 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-05-06T13:56:52.064 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-05-06T13:56:52.064 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:56:52.064 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:56:52.064 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.064 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:56:52.064 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:56:52.064 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:51 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:52.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: Detected new or changed devices on smithi170 2024-05-06T13:56:52.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:56:52.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:56:52.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:56:52.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all crash 2024-05-06T13:56:52.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all mds 2024-05-06T13:56:52.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all rbd-mirror 2024-05-06T13:56:52.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-05-06T13:56:52.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-05-06T13:56:52.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-05-06T13:56:52.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-05-06T13:56:52.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-05-06T13:56:52.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-05-06T13:56:52.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-05-06T13:56:52.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:56:52.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-05-06T13:56:52.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-05-06T13:56:52.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-05-06T13:56:52.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:56:52.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-05-06T13:56:52.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-05-06T13:56:52.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-05-06T13:56:52.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-05-06T13:56:52.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-05-06T13:56:52.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-05-06T13:56:52.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-05-06T13:56:52.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-05-06T13:56:52.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-05-06T13:56:52.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:56:52.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-05-06T13:56:52.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-05-06T13:56:52.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-05-06T13:56:52.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-05-06T13:56:52.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-05-06T13:56:52.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-05-06T13:56:52.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-05-06T13:56:52.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:56:52.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:56:52.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:56:52.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:56:52.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:56:52.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:56:52.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-05-06T13:56:52.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-05-06T13:56:52.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:56:52.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:56:52.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:52.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:56:52.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:56:52.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:51 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:53.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:52 smithi077 ceph-mon[27166]: Upgrade: Setting container_image for all cephfs-mirror 2024-05-06T13:56:53.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:52 smithi077 ceph-mon[27166]: Upgrade: Setting container_image for all ceph-exporter 2024-05-06T13:56:53.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:52 smithi077 ceph-mon[27166]: Upgrade: Setting container_image for all nfs 2024-05-06T13:56:53.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:52 smithi077 ceph-mon[27166]: Upgrade: Setting container_image for all nvmeof 2024-05-06T13:56:53.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:52 smithi077 ceph-mon[27166]: Upgrade: Setting container_image for all node-exporter 2024-05-06T13:56:53.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:52 smithi077 ceph-mon[27166]: Upgrade: Setting container_image for all prometheus 2024-05-06T13:56:53.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:52 smithi077 ceph-mon[27166]: Upgrade: Setting container_image for all alertmanager 2024-05-06T13:56:53.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:52 smithi077 ceph-mon[27166]: Upgrade: Setting container_image for all grafana 2024-05-06T13:56:53.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:52 smithi077 ceph-mon[27166]: Upgrade: Setting container_image for all loki 2024-05-06T13:56:53.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:52 smithi077 ceph-mon[27166]: Upgrade: Setting container_image for all promtail 2024-05-06T13:56:53.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:52 smithi077 ceph-mon[27166]: Upgrade: Finalizing container_image settings 2024-05-06T13:56:53.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:52 smithi077 ceph-mon[27166]: Upgrade: Complete! 2024-05-06T13:56:53.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:52 smithi077 ceph-mon[22428]: Upgrade: Setting container_image for all cephfs-mirror 2024-05-06T13:56:53.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:52 smithi077 ceph-mon[22428]: Upgrade: Setting container_image for all ceph-exporter 2024-05-06T13:56:53.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:52 smithi077 ceph-mon[22428]: Upgrade: Setting container_image for all nfs 2024-05-06T13:56:53.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:52 smithi077 ceph-mon[22428]: Upgrade: Setting container_image for all nvmeof 2024-05-06T13:56:53.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:52 smithi077 ceph-mon[22428]: Upgrade: Setting container_image for all node-exporter 2024-05-06T13:56:53.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:52 smithi077 ceph-mon[22428]: Upgrade: Setting container_image for all prometheus 2024-05-06T13:56:53.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:52 smithi077 ceph-mon[22428]: Upgrade: Setting container_image for all alertmanager 2024-05-06T13:56:53.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:52 smithi077 ceph-mon[22428]: Upgrade: Setting container_image for all grafana 2024-05-06T13:56:53.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:52 smithi077 ceph-mon[22428]: Upgrade: Setting container_image for all loki 2024-05-06T13:56:53.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:52 smithi077 ceph-mon[22428]: Upgrade: Setting container_image for all promtail 2024-05-06T13:56:53.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:52 smithi077 ceph-mon[22428]: Upgrade: Finalizing container_image settings 2024-05-06T13:56:53.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:52 smithi077 ceph-mon[22428]: Upgrade: Complete! 2024-05-06T13:56:53.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:52 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all cephfs-mirror 2024-05-06T13:56:53.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:52 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all ceph-exporter 2024-05-06T13:56:53.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:52 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all nfs 2024-05-06T13:56:53.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:52 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all nvmeof 2024-05-06T13:56:53.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:52 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all node-exporter 2024-05-06T13:56:53.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:52 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all prometheus 2024-05-06T13:56:53.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:52 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all alertmanager 2024-05-06T13:56:53.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:52 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all grafana 2024-05-06T13:56:53.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:52 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all loki 2024-05-06T13:56:53.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:52 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all promtail 2024-05-06T13:56:53.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:52 smithi170 ceph-mon[65866]: Upgrade: Finalizing container_image settings 2024-05-06T13:56:53.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:52 smithi170 ceph-mon[65866]: Upgrade: Complete! 2024-05-06T13:56:54.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:56:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:56:53] "GET /metrics HTTP/1.1" 200 38006 "" "Prometheus/2.43.0" 2024-05-06T13:56:54.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:53 smithi077 ceph-mon[22428]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:54.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:53 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:54.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:53 smithi077 ceph-mon[27166]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:54.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:53 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:54.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:53 smithi170 ceph-mon[65866]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:54.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:53 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:56:56.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:55 smithi077 ceph-mon[22428]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:56:56.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:55 smithi077 ceph-mon[22428]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:56.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:55 smithi077 ceph-mon[27166]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:56:56.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:55 smithi077 ceph-mon[27166]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:56.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:55 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:56:56.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:55 smithi170 ceph-mon[65866]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:56:58.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:57 smithi077 ceph-mon[22428]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:58.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:57 smithi077 ceph-mon[27166]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:56:58.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:57 smithi170 ceph-mon[65866]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:57:00.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:56:59 smithi077 ceph-mon[22428]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:57:00.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:56:59 smithi077 ceph-mon[27166]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:57:00.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:56:59 smithi170 ceph-mon[65866]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:57:02.105 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:01 smithi077 ceph-mon[22428]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:57:02.106 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:01 smithi077 ceph-mon[27166]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:57:02.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:01 smithi170 ceph-mon[65866]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:57:03.128 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:02 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:57:03.128 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:02 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:57:03.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:02 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:57:03.802 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:57:03] "GET /metrics HTTP/1.1" 200 38003 "" "Prometheus/2.43.0" 2024-05-06T13:57:04.267 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:03 smithi077 ceph-mon[22428]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:57:04.267 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:03 smithi077 ceph-mon[27166]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:57:04.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:03 smithi170 ceph-mon[65866]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:57:06.305 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:05 smithi077 ceph-mon[22428]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:57:06.306 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:05 smithi077 ceph-mon[22428]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:57:06.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:05 smithi077 ceph-mon[27166]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:57:06.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:05 smithi077 ceph-mon[27166]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:57:06.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:05 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:57:06.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:05 smithi170 ceph-mon[65866]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:57:08.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:07 smithi077 ceph-mon[22428]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:57:08.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:07 smithi077 ceph-mon[27166]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:57:08.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:07 smithi170 ceph-mon[65866]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:57:09.781 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch ps' 2024-05-06T13:57:10.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:09 smithi077 ceph-mon[22428]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:57:10.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:09 smithi077 ceph-mon[27166]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:57:10.129 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:09 smithi170 ceph-mon[65866]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:57:11.009 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:10 smithi077 ceph-mon[22428]: from='client.25771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:57:11.010 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:10 smithi077 ceph-mon[27166]: from='client.25771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:57:11.142 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:10 smithi170 ceph-mon[65866]: from='client.25771 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:57:11.729 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T13:57:11.729 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (119s) 114s ago 24m 17.1M - 0.25.0 c8568f914cd2 997175e3e64e 2024-05-06T13:57:11.729 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (96s) 25s ago 24m 77.1M - 9.4.12 2bacad6d85d8 140bb9a2760a 2024-05-06T13:57:11.729 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (2m) 114s ago 23m 52.4M - 3.5 e1d6a67b021e 9353a303fb55 2024-05-06T13:57:11.729 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283,8765 running (3m) 25s ago 28m 453M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 125521be53a3 2024-05-06T13:57:11.729 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (12m) 114s ago 30m 504M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T13:57:11.729 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (30m) 114s ago 30m 78.0M 2048M 17.2.0 e1d6a67b021e ac46f91694f8 2024-05-06T13:57:11.729 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (30s) 25s ago 29m 18.3M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 61cd48b3d0ff 2024-05-06T13:57:11.729 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (29m) 114s ago 29m 57.2M 2048M 17.2.0 e1d6a67b021e 83af336ef762 2024-05-06T13:57:11.729 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (2m) 114s ago 25m 14.4M - 1.5.0 0da6a335fe13 67f45fa4968b 2024-05-06T13:57:11.730 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (2m) 25s ago 25m 16.1M - 1.5.0 0da6a335fe13 02081b71a5dd 2024-05-06T13:57:11.730 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (28m) 114s ago 28m 54.4M 2992M 17.2.0 e1d6a67b021e 2ccee2bb6e96 2024-05-06T13:57:11.730 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (28m) 114s ago 28m 58.5M 2992M 17.2.0 e1d6a67b021e 1d9e6609b585 2024-05-06T13:57:11.730 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (27m) 114s ago 27m 52.4M 2992M 17.2.0 e1d6a67b021e 37a61d295111 2024-05-06T13:57:11.730 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (27m) 114s ago 27m 52.9M 2992M 17.2.0 e1d6a67b021e 8feb7c077aaa 2024-05-06T13:57:11.730 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (27m) 25s ago 27m 58.4M 3248M 17.2.0 e1d6a67b021e f865acc44a06 2024-05-06T13:57:11.730 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (26m) 25s ago 26m 52.5M 3248M 17.2.0 e1d6a67b021e a55bd4122f6f 2024-05-06T13:57:11.730 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (26m) 25s ago 26m 51.3M 3248M 17.2.0 e1d6a67b021e 362938a0b357 2024-05-06T13:57:11.730 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (25m) 25s ago 25m 53.9M 3248M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T13:57:11.730 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (2m) 25s ago 25m 60.0M - 2.43.0 a07b618ecd1d 10d5c513bb9b 2024-05-06T13:57:11.730 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (23m) 114s ago 23m 77.3M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T13:57:11.731 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (23m) 25s ago 23m 76.9M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T13:57:12.288 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:11 smithi170 ceph-mon[65866]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:57:12.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:11 smithi077 ceph-mon[22428]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:57:12.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:11 smithi077 ceph-mon[27166]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:57:12.377 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph versions | jq -e '"'"'.mon | length == 2'"'"'' 2024-05-06T13:57:12.990 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:12 smithi077 ceph-mon[22428]: from='client.25777 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:57:12.990 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:12 smithi077 ceph-mon[27166]: from='client.25777 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:57:13.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:12 smithi170 ceph-mon[65866]: from='client.25777 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:57:13.873 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:57:13] "GET /metrics HTTP/1.1" 200 38003 "" "Prometheus/2.43.0" 2024-05-06T13:57:14.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:13 smithi077 ceph-mon[22428]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:57:14.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:13 smithi077 ceph-mon[27166]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:57:14.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:13 smithi170 ceph-mon[65866]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:57:14.396 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T13:57:15.011 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:14 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/1829670347' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:57:15.011 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:14 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/1829670347' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:57:15.037 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch upgrade status' 2024-05-06T13:57:15.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:14 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/1829670347' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:57:15.993 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:15 smithi077 ceph-mon[22428]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:57:15.994 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:15 smithi077 ceph-mon[22428]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:57:15.994 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:15 smithi077 ceph-mon[27166]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:57:15.994 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:15 smithi077 ceph-mon[27166]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:57:16.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:15 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:57:16.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:15 smithi170 ceph-mon[65866]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:57:16.981 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:57:16.981 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": null, 2024-05-06T13:57:16.981 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": false, 2024-05-06T13:57:16.981 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "", 2024-05-06T13:57:16.981 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [], 2024-05-06T13:57:16.982 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": null, 2024-05-06T13:57:16.982 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "", 2024-05-06T13:57:16.982 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-05-06T13:57:16.982 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:57:17.666 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph health detail' 2024-05-06T13:57:17.969 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:17 smithi077 ceph-mon[22428]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:57:17.969 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:17 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:57:17.969 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:17 smithi077 ceph-mon[27166]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:57:17.969 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:17 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:57:18.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:17 smithi170 ceph-mon[65866]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:57:18.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:17 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:57:19.229 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:18 smithi077 ceph-mon[22428]: from='client.15888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:57:19.229 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:18 smithi077 ceph-mon[27166]: from='client.15888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:57:19.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:18 smithi170 ceph-mon[65866]: from='client.15888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:57:19.757 INFO:teuthology.orchestra.run.smithi077.stdout:HEALTH_OK 2024-05-06T13:57:20.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:19 smithi077 ceph-mon[22428]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:57:20.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:19 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/1489222363' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T13:57:20.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:19 smithi077 ceph-mon[27166]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:57:20.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:19 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/1489222363' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T13:57:20.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:19 smithi170 ceph-mon[65866]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:57:20.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:19 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/1489222363' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T13:57:20.419 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.y | awk '"'"'{print $2}'"'"')' 2024-05-06T13:57:21.952 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:21 smithi077 ceph-mon[22428]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:57:21.952 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:21 smithi077 ceph-mon[27166]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:57:22.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:21 smithi170 ceph-mon[65866]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:57:23.902 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:57:23] "GET /metrics HTTP/1.1" 200 38005 "" "Prometheus/2.43.0" 2024-05-06T13:57:24.297 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:23 smithi077 ceph-mon[22428]: from='client.25801 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:57:24.297 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:23 smithi077 ceph-mon[22428]: from='client.25807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "daemon_types": "mon", "hosts": "smithi077", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:57:24.297 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:23 smithi077 ceph-mon[22428]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:57:24.297 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:23 smithi077 ceph-mon[27166]: from='client.25801 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:57:24.298 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:23 smithi077 ceph-mon[27166]: from='client.25807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "daemon_types": "mon", "hosts": "smithi077", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:57:24.298 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:23 smithi077 ceph-mon[27166]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:57:24.330 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:23 smithi170 ceph-mon[65866]: from='client.25801 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:57:24.330 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:23 smithi170 ceph-mon[65866]: from='client.25807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "daemon_types": "mon", "hosts": "smithi077", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:57:24.330 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:23 smithi170 ceph-mon[65866]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:57:24.369 INFO:teuthology.orchestra.run.smithi077.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:57:24.963 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-05-06T13:57:25.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:25 smithi077 ceph-mon[27166]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:57:25.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:25 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:57:25.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:25 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:57:25.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:25 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:57:25.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:25 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:57:25.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:25 smithi077 ceph-mon[27166]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:57:25.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:25 smithi077 ceph-mon[27166]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:57:25.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:25 smithi077 ceph-mon[27166]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:57:25.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:25 smithi077 ceph-mon[22428]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:57:25.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:25 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:57:25.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:25 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:57:25.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:25 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:57:25.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:25 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:57:25.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:25 smithi077 ceph-mon[22428]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:57:25.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:25 smithi077 ceph-mon[22428]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:57:25.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:25 smithi077 ceph-mon[22428]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:57:25.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:25 smithi170 ceph-mon[65866]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:57:25.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:25 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:57:25.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:25 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:57:25.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:25 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:57:25.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:25 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:57:25.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:25 smithi170 ceph-mon[65866]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:57:25.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:25 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:57:25.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:25 smithi170 ceph-mon[65866]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:57:27.217 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:27 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:57:27.218 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:27 smithi077 ceph-mon[22428]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (squid) 2024-05-06T13:57:27.218 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:27 smithi077 ceph-mon[22428]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T13:57:27.218 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:27 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:57:27.218 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:27 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:57:27.218 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:27 smithi077 ceph-mon[22428]: Upgrade: Setting container_image for all mgr 2024-05-06T13:57:27.218 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:27 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:57:27.218 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:27 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-05-06T13:57:27.218 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:27 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-05-06T13:57:27.218 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:27 smithi077 ceph-mon[22428]: Upgrade: It appears safe to stop mon.a 2024-05-06T13:57:27.218 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:27 smithi077 ceph-mon[22428]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:57:27.219 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:27 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:57:27.219 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:27 smithi077 ceph-mon[27166]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (squid) 2024-05-06T13:57:27.219 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:27 smithi077 ceph-mon[27166]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T13:57:27.219 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:27 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:57:27.219 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:27 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:57:27.219 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:27 smithi077 ceph-mon[27166]: Upgrade: Setting container_image for all mgr 2024-05-06T13:57:27.219 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:27 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:57:27.219 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:27 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-05-06T13:57:27.219 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:27 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-05-06T13:57:27.219 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:27 smithi077 ceph-mon[27166]: Upgrade: It appears safe to stop mon.a 2024-05-06T13:57:27.219 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:27 smithi077 ceph-mon[27166]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:57:27.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:27 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:57:27.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:27 smithi170 ceph-mon[65866]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (squid) 2024-05-06T13:57:27.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:27 smithi170 ceph-mon[65866]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T13:57:27.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:27 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:57:27.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:27 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:57:27.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:27 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all mgr 2024-05-06T13:57:27.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:27 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:57:27.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:27 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-05-06T13:57:27.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:27 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-05-06T13:57:27.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:27 smithi170 ceph-mon[65866]: Upgrade: It appears safe to stop mon.a 2024-05-06T13:57:27.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:27 smithi170 ceph-mon[65866]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:57:28.012 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T13:57:28.674 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T13:57:28.674 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (2m) 2m ago 24m 17.1M - 0.25.0 c8568f914cd2 997175e3e64e 2024-05-06T13:57:28.674 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (113s) 42s ago 24m 77.1M - 9.4.12 2bacad6d85d8 140bb9a2760a 2024-05-06T13:57:28.674 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (2m) 2m ago 23m 52.4M - 3.5 e1d6a67b021e 9353a303fb55 2024-05-06T13:57:28.674 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283,8765 running (3m) 42s ago 29m 453M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 125521be53a3 2024-05-06T13:57:28.674 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (13m) 2m ago 30m 504M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T13:57:28.674 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (30m) 2m ago 30m 78.0M 2048M 17.2.0 e1d6a67b021e ac46f91694f8 2024-05-06T13:57:28.674 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (47s) 42s ago 29m 18.3M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 61cd48b3d0ff 2024-05-06T13:57:28.675 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (29m) 2m ago 29m 57.2M 2048M 17.2.0 e1d6a67b021e 83af336ef762 2024-05-06T13:57:28.675 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (3m) 2m ago 25m 14.4M - 1.5.0 0da6a335fe13 67f45fa4968b 2024-05-06T13:57:28.675 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (3m) 42s ago 25m 16.1M - 1.5.0 0da6a335fe13 02081b71a5dd 2024-05-06T13:57:28.675 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (28m) 2m ago 28m 54.4M 2992M 17.2.0 e1d6a67b021e 2ccee2bb6e96 2024-05-06T13:57:28.675 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (28m) 2m ago 28m 58.5M 2992M 17.2.0 e1d6a67b021e 1d9e6609b585 2024-05-06T13:57:28.675 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (28m) 2m ago 28m 52.4M 2992M 17.2.0 e1d6a67b021e 37a61d295111 2024-05-06T13:57:28.675 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (27m) 2m ago 27m 52.9M 2992M 17.2.0 e1d6a67b021e 8feb7c077aaa 2024-05-06T13:57:28.675 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (27m) 42s ago 27m 58.4M 3248M 17.2.0 e1d6a67b021e f865acc44a06 2024-05-06T13:57:28.675 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (26m) 42s ago 26m 52.5M 3248M 17.2.0 e1d6a67b021e a55bd4122f6f 2024-05-06T13:57:28.675 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (26m) 42s ago 26m 51.3M 3248M 17.2.0 e1d6a67b021e 362938a0b357 2024-05-06T13:57:28.675 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (26m) 42s ago 26m 53.9M 3248M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T13:57:28.675 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (2m) 42s ago 25m 60.0M - 2.43.0 a07b618ecd1d 10d5c513bb9b 2024-05-06T13:57:28.675 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (23m) 2m ago 23m 77.3M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T13:57:28.676 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (23m) 42s ago 23m 76.9M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T13:57:29.087 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:57:29.087 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-05-06T13:57:29.087 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-05-06T13:57:29.087 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 1 2024-05-06T13:57:29.087 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:57:29.088 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-05-06T13:57:29.088 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T13:57:29.088 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:57:29.088 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-05-06T13:57:29.088 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-06T13:57:29.088 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:57:29.088 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": {}, 2024-05-06T13:57:29.088 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw": { 2024-05-06T13:57:29.088 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T13:57:29.088 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:57:29.088 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-05-06T13:57:29.088 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-05-06T13:57:29.088 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T13:57:29.088 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T13:57:29.088 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:57:29.432 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:57:29.432 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T13:57:29.432 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-05-06T13:57:29.433 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi077", 2024-05-06T13:57:29.433 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [], 2024-05-06T13:57:29.433 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "0/2 daemons upgraded", 2024-05-06T13:57:29.433 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "Currently upgrading mon daemons", 2024-05-06T13:57:29.433 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-05-06T13:57:29.433 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:57:30.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:29 smithi077 ceph-mon[22428]: from='client.25813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:57:30.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:29 smithi077 ceph-mon[22428]: from='client.25819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:57:30.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:29 smithi077 ceph-mon[22428]: from='client.25825 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:57:30.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:29 smithi077 ceph-mon[22428]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:57:30.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:29 smithi077 ceph-mon[22428]: Upgrade: Updating mon.a 2024-05-06T13:57:30.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:29 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:57:30.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:29 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:57:30.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:29 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T13:57:30.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:29 smithi077 ceph-mon[22428]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:57:30.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:29 smithi077 ceph-mon[22428]: Deploying daemon mon.a on smithi077 2024-05-06T13:57:30.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:29 smithi077 ceph-mon[22428]: from='client.? 172.21.15.77:0/313450589' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:57:30.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:29 smithi077 ceph-mon[27166]: from='client.25813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:57:30.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:29 smithi077 ceph-mon[27166]: from='client.25819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:57:30.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:29 smithi077 ceph-mon[27166]: from='client.25825 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:57:30.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:29 smithi077 ceph-mon[27166]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:57:30.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:29 smithi077 ceph-mon[27166]: Upgrade: Updating mon.a 2024-05-06T13:57:30.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:29 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:57:30.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:29 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:57:30.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:29 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T13:57:30.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:29 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:57:30.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:29 smithi077 ceph-mon[27166]: Deploying daemon mon.a on smithi077 2024-05-06T13:57:30.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:29 smithi077 ceph-mon[27166]: from='client.? 172.21.15.77:0/313450589' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:57:30.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:29 smithi170 ceph-mon[65866]: from='client.25813 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:57:30.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:29 smithi170 ceph-mon[65866]: from='client.25819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:57:30.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:29 smithi170 ceph-mon[65866]: from='client.25825 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:57:30.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:29 smithi170 ceph-mon[65866]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:57:30.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:29 smithi170 ceph-mon[65866]: Upgrade: Updating mon.a 2024-05-06T13:57:30.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:29 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' 2024-05-06T13:57:30.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:29 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:57:30.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:29 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T13:57:30.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:29 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:57:30.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:29 smithi170 ceph-mon[65866]: Deploying daemon mon.a on smithi077 2024-05-06T13:57:30.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:29 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/313450589' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:57:31.002 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:30 smithi077 systemd[1]: Stopping Ceph mon.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:57:31.002 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:30 smithi077 ceph-mon[22428]: from='client.25837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:57:31.003 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:30 smithi077 ceph-mon[27166]: from='client.25837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:57:31.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:31 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-a[22424]: 2024-05-06T13:57:31.000+0000 7f63d7bfb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T13:57:31.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:31 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-a[22424]: 2024-05-06T13:57:31.000+0000 7f63d7bfb700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-05-06T13:57:31.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:31 smithi077 podman[84070]: 2024-05-06 13:57:31.12375339 +0000 UTC m=+0.233408950 container died ac46f91694f871a4c86d9233330db0effb6a0b7cdd415dbb8c13ed18d2c6e0a7 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-a, name=centos-stream, io.openshift.tags=base centos centos-stream, version=8, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , ceph=True, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-05-06T13:57:31.303 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:31 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:57:31] ENGINE Bus STOPPING 2024-05-06T13:57:31.303 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:31 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:57:31] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-06T13:57:31.303 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:31 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:57:31] ENGINE Bus STOPPED 2024-05-06T13:57:31.303 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:31 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:57:31] ENGINE Bus STARTING 2024-05-06T13:57:31.304 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:31 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:57:31] ENGINE Serving on http://:::9283 2024-05-06T13:57:31.304 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:31 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:57:31] ENGINE Bus STARTED 2024-05-06T13:57:31.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:30 smithi170 ceph-mon[65866]: from='client.25837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:57:31.736 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:31 smithi077 podman[84070]: 2024-05-06 13:57:31.542890656 +0000 UTC m=+0.652546216 container cleanup ac46f91694f871a4c86d9233330db0effb6a0b7cdd415dbb8c13ed18d2c6e0a7 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-a, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, ceph=True, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , distribution-scope=public, io.openshift.tags=base centos centos-stream, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-05-06T13:57:31.736 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:31 smithi077 bash[84070]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-a 2024-05-06T13:57:32.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:31 smithi077 podman[84095]: 2024-05-06 13:57:31.736178222 +0000 UTC m=+0.604768711 container remove ac46f91694f871a4c86d9233330db0effb6a0b7cdd415dbb8c13ed18d2c6e0a7 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-a, release=754, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, version=8, ceph=True, distribution-scope=public, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8) 2024-05-06T13:57:32.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:31 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-06T13:57:32.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:31 smithi077 systemd[1]: Stopped Ceph mon.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:57:32.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:31 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mon.a.service: Consumed 25.795s CPU time. 2024-05-06T13:57:32.378 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 systemd[1]: Starting Ceph mon.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:57:32.632 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 podman[84227]: 2024-05-06 13:57:32.281909632 +0000 UTC m=+0.018384426 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:57:32.632 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 podman[84227]: 2024-05-06 13:57:32.404897457 +0000 UTC m=+0.141372277 container create 5e50731ccbe87044c3f1a3e8f154130cf0f595ef2d3249272436cba1a9a5bf51 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-a, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-70ce953) 2024-05-06T13:57:33.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 podman[84227]: 2024-05-06 13:57:32.632564097 +0000 UTC m=+0.369038876 container init 5e50731ccbe87044c3f1a3e8f154130cf0f595ef2d3249272436cba1a9a5bf51 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-a, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, ceph=True, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T13:57:33.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 podman[84227]: 2024-05-06 13:57:32.635634822 +0000 UTC m=+0.372109607 container start 5e50731ccbe87044c3f1a3e8f154130cf0f595ef2d3249272436cba1a9a5bf51 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-05-06T13:57:33.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: set uid:gid to 167:167 (ceph:ceph) 2024-05-06T13:57:33.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev), process ceph-mon, pid 2 2024-05-06T13:57:33.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: pidfile_write: ignore empty --pid-file 2024-05-06T13:57:33.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: load: jerasure load: lrc 2024-05-06T13:57:33.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: RocksDB version: 7.9.2 2024-05-06T13:57:33.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Git sha 0 2024-05-06T13:57:33.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Compile date 2024-05-03 15:08:47 2024-05-06T13:57:33.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: DB SUMMARY 2024-05-06T13:57:33.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: DB Session ID: UNROIR2T2EFU0CSBNZYZ 2024-05-06T13:57:33.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: CURRENT file: CURRENT 2024-05-06T13:57:33.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: IDENTITY file: IDENTITY 2024-05-06T13:57:33.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: MANIFEST file: MANIFEST-000015 size: 3088 Bytes 2024-05-06T13:57:33.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000063.sst 2024-05-06T13:57:33.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000061.log size: 187032 ; 2024-05-06T13:57:33.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.error_if_exists: 0 2024-05-06T13:57:33.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.create_if_missing: 0 2024-05-06T13:57:33.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.paranoid_checks: 1 2024-05-06T13:57:33.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-06T13:57:33.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-06T13:57:33.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-06T13:57:33.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.env: 0x561cd9f19c60 2024-05-06T13:57:33.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.fs: PosixFileSystem 2024-05-06T13:57:33.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.info_log: 0x561cdc51fac0 2024-05-06T13:57:33.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.max_file_opening_threads: 16 2024-05-06T13:57:33.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.statistics: (nil) 2024-05-06T13:57:33.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.use_fsync: 0 2024-05-06T13:57:33.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.max_log_file_size: 0 2024-05-06T13:57:33.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-06T13:57:33.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-06T13:57:33.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.keep_log_file_num: 1000 2024-05-06T13:57:33.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.recycle_log_file_num: 0 2024-05-06T13:57:33.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.allow_fallocate: 1 2024-05-06T13:57:33.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.allow_mmap_reads: 0 2024-05-06T13:57:33.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.allow_mmap_writes: 0 2024-05-06T13:57:33.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.use_direct_reads: 0 2024-05-06T13:57:33.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-06T13:57:33.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.create_missing_column_families: 0 2024-05-06T13:57:33.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.db_log_dir: 2024-05-06T13:57:33.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.wal_dir: 2024-05-06T13:57:33.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-06T13:57:33.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-06T13:57:33.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-06T13:57:33.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-06T13:57:33.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-06T13:57:33.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-06T13:57:33.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.advise_random_on_open: 1 2024-05-06T13:57:33.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.db_write_buffer_size: 0 2024-05-06T13:57:33.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.write_buffer_manager: 0x561cdc4f2820 2024-05-06T13:57:33.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-06T13:57:33.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-06T13:57:33.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-06T13:57:33.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.rate_limiter: (nil) 2024-05-06T13:57:33.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-06T13:57:33.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.wal_recovery_mode: 2 2024-05-06T13:57:33.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.enable_thread_tracking: 0 2024-05-06T13:57:33.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.enable_pipelined_write: 0 2024-05-06T13:57:33.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.unordered_write: 0 2024-05-06T13:57:33.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-06T13:57:33.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-06T13:57:33.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-06T13:57:33.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-06T13:57:33.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.row_cache: None 2024-05-06T13:57:33.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.wal_filter: None 2024-05-06T13:57:33.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-06T13:57:33.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.allow_ingest_behind: 0 2024-05-06T13:57:33.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.two_write_queues: 0 2024-05-06T13:57:33.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.manual_wal_flush: 0 2024-05-06T13:57:33.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.wal_compression: 0 2024-05-06T13:57:33.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.atomic_flush: 0 2024-05-06T13:57:33.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-06T13:57:33.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-06T13:57:33.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-06T13:57:33.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.log_readahead_size: 0 2024-05-06T13:57:33.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-06T13:57:33.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.best_efforts_recovery: 0 2024-05-06T13:57:33.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-06T13:57:33.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-06T13:57:33.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.allow_data_in_errors: 0 2024-05-06T13:57:33.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.db_host_id: __hostname__ 2024-05-06T13:57:33.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-06T13:57:33.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.max_background_jobs: 2 2024-05-06T13:57:33.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.max_background_compactions: -1 2024-05-06T13:57:33.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.max_subcompactions: 1 2024-05-06T13:57:33.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-06T13:57:33.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-06T13:57:33.060 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-06T13:57:33.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.max_total_wal_size: 0 2024-05-06T13:57:33.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-06T13:57:33.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-06T13:57:33.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-06T13:57:33.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-06T13:57:33.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.max_open_files: -1 2024-05-06T13:57:33.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.bytes_per_sync: 0 2024-05-06T13:57:33.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-06T13:57:33.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-06T13:57:33.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.compaction_readahead_size: 0 2024-05-06T13:57:33.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.max_background_flushes: -1 2024-05-06T13:57:33.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Compression algorithms supported: 2024-05-06T13:57:33.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-06T13:57:33.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: kZSTD supported: 0 2024-05-06T13:57:33.061 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: kXpressCompression supported: 0 2024-05-06T13:57:33.062 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: kLZ4HCCompression supported: 1 2024-05-06T13:57:33.062 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: kZlibCompression supported: 1 2024-05-06T13:57:33.062 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: kSnappyCompression supported: 1 2024-05-06T13:57:33.062 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: kLZ4Compression supported: 1 2024-05-06T13:57:33.062 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: kBZip2Compression supported: 0 2024-05-06T13:57:33.062 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-06T13:57:33.062 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-06T13:57:33.062 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-05-06T13:57:33.062 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-06T13:57:33.062 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-06T13:57:33.062 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.merge_operator: 2024-05-06T13:57:33.062 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.compaction_filter: None 2024-05-06T13:57:33.062 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.compaction_filter_factory: None 2024-05-06T13:57:33.062 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.sst_partitioner_factory: None 2024-05-06T13:57:33.062 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-06T13:57:33.063 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-06T13:57:33.063 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561cdc51fbc0) 2024-05-06T13:57:33.063 INFO:journalctl@ceph.mon.a.smithi077.stdout: cache_index_and_filter_blocks: 1 2024-05-06T13:57:33.063 INFO:journalctl@ceph.mon.a.smithi077.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-06T13:57:33.063 INFO:journalctl@ceph.mon.a.smithi077.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-06T13:57:33.063 INFO:journalctl@ceph.mon.a.smithi077.stdout: pin_top_level_index_and_filter: 1 2024-05-06T13:57:33.063 INFO:journalctl@ceph.mon.a.smithi077.stdout: index_type: 0 2024-05-06T13:57:33.063 INFO:journalctl@ceph.mon.a.smithi077.stdout: data_block_index_type: 0 2024-05-06T13:57:33.063 INFO:journalctl@ceph.mon.a.smithi077.stdout: index_shortening: 1 2024-05-06T13:57:33.063 INFO:journalctl@ceph.mon.a.smithi077.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-06T13:57:33.063 INFO:journalctl@ceph.mon.a.smithi077.stdout: checksum: 4 2024-05-06T13:57:33.063 INFO:journalctl@ceph.mon.a.smithi077.stdout: no_block_cache: 0 2024-05-06T13:57:33.063 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_cache: 0x561cdc523090 2024-05-06T13:57:33.063 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_cache_name: BinnedLRUCache 2024-05-06T13:57:33.063 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_cache_options: 2024-05-06T13:57:33.064 INFO:journalctl@ceph.mon.a.smithi077.stdout: capacity : 536870912 2024-05-06T13:57:33.064 INFO:journalctl@ceph.mon.a.smithi077.stdout: num_shard_bits : 4 2024-05-06T13:57:33.064 INFO:journalctl@ceph.mon.a.smithi077.stdout: strict_capacity_limit : 0 2024-05-06T13:57:33.064 INFO:journalctl@ceph.mon.a.smithi077.stdout: high_pri_pool_ratio: 0.000 2024-05-06T13:57:33.064 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_cache_compressed: (nil) 2024-05-06T13:57:33.064 INFO:journalctl@ceph.mon.a.smithi077.stdout: persistent_cache: (nil) 2024-05-06T13:57:33.064 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_size: 4096 2024-05-06T13:57:33.064 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_size_deviation: 10 2024-05-06T13:57:33.064 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_restart_interval: 16 2024-05-06T13:57:33.064 INFO:journalctl@ceph.mon.a.smithi077.stdout: index_block_restart_interval: 1 2024-05-06T13:57:33.064 INFO:journalctl@ceph.mon.a.smithi077.stdout: metadata_block_size: 4096 2024-05-06T13:57:33.064 INFO:journalctl@ceph.mon.a.smithi077.stdout: partition_filters: 0 2024-05-06T13:57:33.064 INFO:journalctl@ceph.mon.a.smithi077.stdout: use_delta_encoding: 1 2024-05-06T13:57:33.064 INFO:journalctl@ceph.mon.a.smithi077.stdout: filter_policy: bloomfilter 2024-05-06T13:57:33.064 INFO:journalctl@ceph.mon.a.smithi077.stdout: whole_key_filtering: 1 2024-05-06T13:57:33.065 INFO:journalctl@ceph.mon.a.smithi077.stdout: verify_compression: 0 2024-05-06T13:57:33.065 INFO:journalctl@ceph.mon.a.smithi077.stdout: read_amp_bytes_per_bit: 0 2024-05-06T13:57:33.065 INFO:journalctl@ceph.mon.a.smithi077.stdout: format_version: 5 2024-05-06T13:57:33.065 INFO:journalctl@ceph.mon.a.smithi077.stdout: enable_index_compression: 1 2024-05-06T13:57:33.065 INFO:journalctl@ceph.mon.a.smithi077.stdout: block_align: 0 2024-05-06T13:57:33.065 INFO:journalctl@ceph.mon.a.smithi077.stdout: max_auto_readahead_size: 262144 2024-05-06T13:57:33.065 INFO:journalctl@ceph.mon.a.smithi077.stdout: prepopulate_block_cache: 0 2024-05-06T13:57:33.065 INFO:journalctl@ceph.mon.a.smithi077.stdout: initial_auto_readahead_size: 8192 2024-05-06T13:57:33.065 INFO:journalctl@ceph.mon.a.smithi077.stdout: num_file_reads_for_auto_readahead: 2 2024-05-06T13:57:33.065 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.write_buffer_size: 33554432 2024-05-06T13:57:33.065 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.max_write_buffer_number: 2 2024-05-06T13:57:33.065 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.compression: NoCompression 2024-05-06T13:57:33.065 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.bottommost_compression: Disabled 2024-05-06T13:57:33.065 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.prefix_extractor: nullptr 2024-05-06T13:57:33.065 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-06T13:57:33.065 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.num_levels: 7 2024-05-06T13:57:33.066 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-06T13:57:33.066 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-06T13:57:33.066 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-06T13:57:33.066 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-06T13:57:33.066 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-06T13:57:33.066 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-06T13:57:33.066 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-06T13:57:33.066 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-06T13:57:33.066 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-06T13:57:33.066 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-06T13:57:33.066 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-06T13:57:33.066 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-06T13:57:33.066 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-06T13:57:33.066 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.compression_opts.level: 32767 2024-05-06T13:57:33.066 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.compression_opts.strategy: 0 2024-05-06T13:57:33.067 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-06T13:57:33.067 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-06T13:57:33.067 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-06T13:57:33.067 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-06T13:57:33.067 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.compression_opts.enabled: false 2024-05-06T13:57:33.067 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-06T13:57:33.067 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-06T13:57:33.067 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-06T13:57:33.067 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-06T13:57:33.067 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.target_file_size_base: 67108864 2024-05-06T13:57:33.067 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-06T13:57:33.067 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-06T13:57:33.067 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-06T13:57:33.067 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-06T13:57:33.067 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-06T13:57:33.067 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-06T13:57:33.068 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-06T13:57:33.068 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-06T13:57:33.068 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-06T13:57:33.068 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-06T13:57:33.068 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-06T13:57:33.068 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-06T13:57:33.068 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-06T13:57:33.068 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-06T13:57:33.068 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.arena_block_size: 1048576 2024-05-06T13:57:33.068 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-06T13:57:33.068 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-06T13:57:33.068 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.disable_auto_compactions: 0 2024-05-06T13:57:33.068 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-06T13:57:33.068 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-06T13:57:33.069 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-06T13:57:33.069 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-06T13:57:33.069 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-06T13:57:33.069 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-06T13:57:33.069 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-06T13:57:33.069 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-06T13:57:33.069 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-06T13:57:33.069 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-06T13:57:33.069 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.table_properties_collectors: 2024-05-06T13:57:33.069 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.inplace_update_support: 0 2024-05-06T13:57:33.069 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-06T13:57:33.069 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-06T13:57:33.069 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-06T13:57:33.069 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-06T13:57:33.069 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.bloom_locality: 0 2024-05-06T13:57:33.070 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.max_successive_merges: 0 2024-05-06T13:57:33.070 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-06T13:57:33.070 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.paranoid_file_checks: 0 2024-05-06T13:57:33.070 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.force_consistency_checks: 1 2024-05-06T13:57:33.070 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.report_bg_io_stats: 0 2024-05-06T13:57:33.070 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.ttl: 2592000 2024-05-06T13:57:33.070 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-06T13:57:33.070 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-06T13:57:33.070 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-06T13:57:33.070 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.enable_blob_files: false 2024-05-06T13:57:33.070 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.min_blob_size: 0 2024-05-06T13:57:33.070 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.blob_file_size: 268435456 2024-05-06T13:57:33.070 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-06T13:57:33.070 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-06T13:57:33.070 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-06T13:57:33.071 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-06T13:57:33.071 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-06T13:57:33.071 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.blob_file_starting_level: 0 2024-05-06T13:57:33.071 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-06T13:57:33.071 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 63.sst 2024-05-06T13:57:33.071 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-05-06T13:57:33.071 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 65, last_sequence is 29290, log_number is 61,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-06T13:57:33.071 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 61 2024-05-06T13:57:33.071 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: fe598134-8211-495d-9b79-a1c20296cd97 2024-05-06T13:57:33.071 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715003852665361, "job": 1, "event": "recovery_started", "wal_files": [61]} 2024-05-06T13:57:33.071 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #61 mode 2 2024-05-06T13:57:33.071 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715003852667002, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 66, "file_size": 184408, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 29291, "largest_seqno": 29495, "table_properties": {"data_size": 182587, "index_size": 489, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 325, "raw_key_size": 2567, "raw_average_key_size": 24, "raw_value_size": 180356, "raw_average_value_size": 1751, "num_data_blocks": 22, "num_entries": 103, "num_filter_entries": 103, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715003852, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "fe598134-8211-495d-9b79-a1c20296cd97", "db_session_id": "UNROIR2T2EFU0CSBNZYZ", "orig_file_number": 66, "seqno_to_time_mapping": "N/A"}} 2024-05-06T13:57:33.071 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715003852667101, "job": 1, "event": "recovery_finished"} 2024-05-06T13:57:33.071 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: [db/version_set.cc:5047] Creating manifest 68 2024-05-06T13:57:33.072 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-05-06T13:57:33.072 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000061.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-06T13:57:33.072 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x561cdc614000 2024-05-06T13:57:33.072 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: DB pointer 0x561cdc5fe000 2024-05-06T13:57:33.072 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-06T13:57:33.072 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-06T13:57:33.072 INFO:journalctl@ceph.mon.a.smithi077.stdout: ** DB Stats ** 2024-05-06T13:57:33.072 INFO:journalctl@ceph.mon.a.smithi077.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T13:57:33.072 INFO:journalctl@ceph.mon.a.smithi077.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-06T13:57:33.072 INFO:journalctl@ceph.mon.a.smithi077.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-06T13:57:33.072 INFO:journalctl@ceph.mon.a.smithi077.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-06T13:57:33.073 INFO:journalctl@ceph.mon.a.smithi077.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-06T13:57:33.073 INFO:journalctl@ceph.mon.a.smithi077.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-06T13:57:33.073 INFO:journalctl@ceph.mon.a.smithi077.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-06T13:57:33.073 INFO:journalctl@ceph.mon.a.smithi077.stdout: 2024-05-06T13:57:33.073 INFO:journalctl@ceph.mon.a.smithi077.stdout: ** Compaction Stats [default] ** 2024-05-06T13:57:33.073 INFO:journalctl@ceph.mon.a.smithi077.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-06T13:57:33.073 INFO:journalctl@ceph.mon.a.smithi077.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-06T13:57:33.073 INFO:journalctl@ceph.mon.a.smithi077.stdout: L0 1/0 180.09 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-06T13:57:33.073 INFO:journalctl@ceph.mon.a.smithi077.stdout: L6 1/0 9.45 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-05-06T13:57:33.073 INFO:journalctl@ceph.mon.a.smithi077.stdout: Sum 2/0 9.63 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-06T13:57:33.073 INFO:journalctl@ceph.mon.a.smithi077.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-06T13:57:33.073 INFO:journalctl@ceph.mon.a.smithi077.stdout: 2024-05-06T13:57:33.073 INFO:journalctl@ceph.mon.a.smithi077.stdout: ** Compaction Stats [default] ** 2024-05-06T13:57:33.073 INFO:journalctl@ceph.mon.a.smithi077.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-06T13:57:33.073 INFO:journalctl@ceph.mon.a.smithi077.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-06T13:57:33.074 INFO:journalctl@ceph.mon.a.smithi077.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 162.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-06T13:57:33.074 INFO:journalctl@ceph.mon.a.smithi077.stdout: 2024-05-06T13:57:33.074 INFO:journalctl@ceph.mon.a.smithi077.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-06T13:57:33.074 INFO:journalctl@ceph.mon.a.smithi077.stdout: 2024-05-06T13:57:33.074 INFO:journalctl@ceph.mon.a.smithi077.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T13:57:33.074 INFO:journalctl@ceph.mon.a.smithi077.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-06T13:57:33.074 INFO:journalctl@ceph.mon.a.smithi077.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-06T13:57:33.074 INFO:journalctl@ceph.mon.a.smithi077.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-06T13:57:33.074 INFO:journalctl@ceph.mon.a.smithi077.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-06T13:57:33.074 INFO:journalctl@ceph.mon.a.smithi077.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-06T13:57:33.074 INFO:journalctl@ceph.mon.a.smithi077.stdout: Cumulative compaction: 0.00 GB write, 42.02 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T13:57:33.074 INFO:journalctl@ceph.mon.a.smithi077.stdout: Interval compaction: 0.00 GB write, 42.02 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T13:57:33.074 INFO:journalctl@ceph.mon.a.smithi077.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-06T13:57:33.074 INFO:journalctl@ceph.mon.a.smithi077.stdout: Block cache BinnedLRUCache@0x561cdc523090#2 capacity: 512.00 MB usage: 0.95 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-05-06T13:57:33.075 INFO:journalctl@ceph.mon.a.smithi077.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.38 KB,7.15256e-05%) IndexBlock(1,0.58 KB,0.000110269%) Misc(1,0.00 KB,0%) 2024-05-06T13:57:33.075 INFO:journalctl@ceph.mon.a.smithi077.stdout: 2024-05-06T13:57:33.075 INFO:journalctl@ceph.mon.a.smithi077.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-06T13:57:33.075 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: starting mon.a rank 0 at public addrs [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] at bind addrs [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:57:33.075 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: mon.a@-1(???) e3 preinit fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:57:33.075 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: mon.a@-1(???).mds e1 new map 2024-05-06T13:57:33.075 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: mon.a@-1(???).mds e1 print_map 2024-05-06T13:57:33.075 INFO:journalctl@ceph.mon.a.smithi077.stdout: e1 2024-05-06T13:57:33.075 INFO:journalctl@ceph.mon.a.smithi077.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-06T13:57:33.075 INFO:journalctl@ceph.mon.a.smithi077.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-06T13:57:33.075 INFO:journalctl@ceph.mon.a.smithi077.stdout: legacy client fscid: -1 2024-05-06T13:57:33.075 INFO:journalctl@ceph.mon.a.smithi077.stdout: 2024-05-06T13:57:33.075 INFO:journalctl@ceph.mon.a.smithi077.stdout: No filesystems configured 2024-05-06T13:57:33.075 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: mon.a@-1(???).osd e112 crush map has features 3314933000854323200, adjusting msgr requires 2024-05-06T13:57:33.075 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: mon.a@-1(???).osd e112 crush map has features 432629239337189376, adjusting msgr requires 2024-05-06T13:57:33.076 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: mon.a@-1(???).osd e112 crush map has features 432629239337189376, adjusting msgr requires 2024-05-06T13:57:33.076 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: mon.a@-1(???).osd e112 crush map has features 432629239337189376, adjusting msgr requires 2024-05-06T13:57:33.076 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 ceph-mon[84247]: mon.a@-1(???).paxosservice(auth 1..27) refresh upgraded, format 0 -> 3 2024-05-06T13:57:33.076 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 bash[84227]: 5e50731ccbe87044c3f1a3e8f154130cf0f595ef2d3249272436cba1a9a5bf51 2024-05-06T13:57:33.076 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:32 smithi077 systemd[1]: Started Ceph mon.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:57:33.995 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:33 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:57:33] "GET /metrics HTTP/1.1" 200 37935 "" "Prometheus/2.43.0" 2024-05-06T13:57:34.294 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ignoring --setuser ceph since I am not root 2024-05-06T13:57:34.294 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ignoring --setgroup ceph since I am not root 2024-05-06T13:57:34.294 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mgr[60780]: -- 172.21.15.77:0/3245740557 <== mon.1 v2:172.21.15.77:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5645e7a165a0 con 0x5645e79eb000 2024-05-06T13:57:34.295 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[84247]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:57:34.295 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[84247]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:57:34.295 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[84247]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:57:34.295 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[84247]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:57:34.295 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[84247]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:57:34.295 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[84247]: mon.a calling monitor election 2024-05-06T13:57:34.295 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[84247]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-06T13:57:34.295 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[84247]: monmap epoch 3 2024-05-06T13:57:34.295 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[84247]: fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:57:34.295 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[84247]: last_changed 2024-05-06T13:28:05.841521+0000 2024-05-06T13:57:34.295 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[84247]: created 2024-05-06T13:26:34.664804+0000 2024-05-06T13:57:34.295 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[84247]: min_mon_release 17 (quincy) 2024-05-06T13:57:34.295 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[84247]: election_strategy: 1 2024-05-06T13:57:34.295 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[84247]: 0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2024-05-06T13:57:34.296 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[84247]: 1: [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] mon.c 2024-05-06T13:57:34.296 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[84247]: 2: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.b 2024-05-06T13:57:34.296 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[84247]: fsmap 2024-05-06T13:57:34.296 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[84247]: osdmap e112: 8 total, 8 up, 8 in 2024-05-06T13:57:34.296 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[84247]: mgrmap e42: y(active, since 4m), standbys: x 2024-05-06T13:57:34.296 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[84247]: overall HEALTH_OK 2024-05-06T13:57:34.296 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[84247]: from='mgr.25327 ' entity='' 2024-05-06T13:57:34.296 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[84247]: mgrmap e43: y(active, since 4m), standbys: x 2024-05-06T13:57:34.296 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:57:34.296 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:57:34.296 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:57:34.296 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[27166]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:57:34.297 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[27166]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:57:34.297 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[27166]: mon.a calling monitor election 2024-05-06T13:57:34.297 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[27166]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-06T13:57:34.297 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[27166]: monmap epoch 3 2024-05-06T13:57:34.297 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[27166]: fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:57:34.297 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[27166]: last_changed 2024-05-06T13:28:05.841521+0000 2024-05-06T13:57:34.297 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[27166]: created 2024-05-06T13:26:34.664804+0000 2024-05-06T13:57:34.297 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[27166]: min_mon_release 17 (quincy) 2024-05-06T13:57:34.297 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[27166]: election_strategy: 1 2024-05-06T13:57:34.297 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[27166]: 0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2024-05-06T13:57:34.297 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[27166]: 1: [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] mon.c 2024-05-06T13:57:34.297 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[27166]: 2: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.b 2024-05-06T13:57:34.297 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[27166]: fsmap 2024-05-06T13:57:34.297 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[27166]: osdmap e112: 8 total, 8 up, 8 in 2024-05-06T13:57:34.297 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[27166]: mgrmap e42: y(active, since 4m), standbys: x 2024-05-06T13:57:34.297 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[27166]: overall HEALTH_OK 2024-05-06T13:57:34.298 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[27166]: from='mgr.25327 ' entity='' 2024-05-06T13:57:34.298 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:34 smithi077 ceph-mon[27166]: mgrmap e43: y(active, since 4m), standbys: x 2024-05-06T13:57:34.477 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:34 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: ignoring --setuser ceph since I am not root 2024-05-06T13:57:34.478 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:34 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: ignoring --setgroup ceph since I am not root 2024-05-06T13:57:34.478 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:34 smithi170 ceph-mgr[58766]: -- 172.21.15.170:0/2808853715 <== mon.2 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5598666605a0 con 0x559866634800 2024-05-06T13:57:34.478 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:34 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:57:34.293+0000 7fc2dfed7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T13:57:34.478 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:34 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:57:34.476+0000 7fc2dfed7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:57:34.478 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:34 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:57:34.478 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:34 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:57:34.478 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:34 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:57:34.479 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:34 smithi170 ceph-mon[65866]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-05-06T13:57:34.479 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:34 smithi170 ceph-mon[65866]: from='mgr.25327 172.21.15.77:0/1585043725' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:57:34.479 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:34 smithi170 ceph-mon[65866]: mon.a calling monitor election 2024-05-06T13:57:34.479 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:34 smithi170 ceph-mon[65866]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-06T13:57:34.479 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:34 smithi170 ceph-mon[65866]: monmap epoch 3 2024-05-06T13:57:34.479 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:34 smithi170 ceph-mon[65866]: fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:57:34.479 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:34 smithi170 ceph-mon[65866]: last_changed 2024-05-06T13:28:05.841521+0000 2024-05-06T13:57:34.479 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:34 smithi170 ceph-mon[65866]: created 2024-05-06T13:26:34.664804+0000 2024-05-06T13:57:34.479 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:34 smithi170 ceph-mon[65866]: min_mon_release 17 (quincy) 2024-05-06T13:57:34.479 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:34 smithi170 ceph-mon[65866]: election_strategy: 1 2024-05-06T13:57:34.479 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:34 smithi170 ceph-mon[65866]: 0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2024-05-06T13:57:34.479 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:34 smithi170 ceph-mon[65866]: 1: [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] mon.c 2024-05-06T13:57:34.479 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:34 smithi170 ceph-mon[65866]: 2: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.b 2024-05-06T13:57:34.479 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:34 smithi170 ceph-mon[65866]: fsmap 2024-05-06T13:57:34.480 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:34 smithi170 ceph-mon[65866]: osdmap e112: 8 total, 8 up, 8 in 2024-05-06T13:57:34.480 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:34 smithi170 ceph-mon[65866]: mgrmap e42: y(active, since 4m), standbys: x 2024-05-06T13:57:34.480 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:34 smithi170 ceph-mon[65866]: overall HEALTH_OK 2024-05-06T13:57:34.480 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:34 smithi170 ceph-mon[65866]: from='mgr.25327 ' entity='' 2024-05-06T13:57:34.480 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:34 smithi170 ceph-mon[65866]: mgrmap e43: y(active, since 4m), standbys: x 2024-05-06T13:57:34.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:57:34.293+0000 7f5d14331200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-06T13:57:34.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:57:34.482+0000 7f5d14331200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:57:34.826 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:34 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:57:34.579+0000 7fc2dfed7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T13:57:34.900 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:57:34.587+0000 7f5d14331200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-06T13:57:35.302 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:57:34.899+0000 7f5d14331200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T13:57:35.302 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:57:34.989+0000 7f5d14331200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T13:57:35.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:34 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:57:34.886+0000 7fc2dfed7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-06T13:57:35.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:34 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:57:34.971+0000 7fc2dfed7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-06T13:57:36.499 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:36 smithi170 ceph-mon[65866]: mgrmap e44: y(active, since 4m), standbys: x 2024-05-06T13:57:36.500 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:36 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:57:36.154+0000 7fc2dfed7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T13:57:36.500 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:36 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:57:36.325+0000 7fc2dfed7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T13:57:36.528 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:36 smithi077 ceph-mon[84247]: mgrmap e44: y(active, since 4m), standbys: x 2024-05-06T13:57:36.528 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:36 smithi077 ceph-mon[27166]: mgrmap e44: y(active, since 4m), standbys: x 2024-05-06T13:57:36.528 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:36 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:57:36.181+0000 7f5d14331200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-06T13:57:36.529 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:36 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:57:36.353+0000 7f5d14331200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-06T13:57:36.529 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:36 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:57:36.527+0000 7f5d14331200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T13:57:36.790 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:36 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:57:36.498+0000 7fc2dfed7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-06T13:57:37.076 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:36 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:57:36.789+0000 7fc2dfed7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T13:57:37.076 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:36 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:57:36.882+0000 7fc2dfed7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T13:57:37.123 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:36 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:57:36.821+0000 7f5d14331200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-06T13:57:37.123 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:36 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:57:36.914+0000 7f5d14331200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-06T13:57:37.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:37 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:57:37.122+0000 7f5d14331200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T13:57:37.553 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:37 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:57:37.318+0000 7f5d14331200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:57:37.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:37 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:57:37.089+0000 7fc2dfed7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-06T13:57:37.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:37 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:57:37.285+0000 7fc2dfed7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-06T13:57:38.302 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:37 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:57:37.930+0000 7f5d14331200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T13:57:38.320 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:37 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:57:37.893+0000 7fc2dfed7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-06T13:57:38.576 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:57:38.319+0000 7fc2dfed7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T13:57:38.577 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:57:38.416+0000 7fc2dfed7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T13:57:38.671 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:38 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:57:38.362+0000 7f5d14331200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-06T13:57:38.672 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:38 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:57:38.461+0000 7f5d14331200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-06T13:57:39.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:38 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:57:38.670+0000 7f5d14331200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-06T13:57:39.076 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:57:38.621+0000 7fc2dfed7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-06T13:57:39.485 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:39 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:57:39.146+0000 7fc2dfed7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T13:57:39.485 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:39 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:57:39.242+0000 7fc2dfed7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T13:57:39.546 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:39 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:57:39.199+0000 7f5d14331200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-06T13:57:39.546 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:39 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:57:39.295+0000 7f5d14331200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-06T13:57:39.802 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:39 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:57:39.545+0000 7f5d14331200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T13:57:39.802 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:39 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:57:39.681+0000 7f5d14331200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T13:57:39.802 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:39 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:57:39.779+0000 7f5d14331200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T13:57:39.826 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:39 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:57:39.484+0000 7fc2dfed7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-06T13:57:39.826 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:39 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:57:39.621+0000 7fc2dfed7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-06T13:57:39.826 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:39 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:57:39.718+0000 7fc2dfed7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-06T13:57:40.302 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:39 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:57:39.899+0000 7f5d14331200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T13:57:40.302 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:39 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:57:39.996+0000 7f5d14331200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T13:57:40.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:39 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:57:39.833+0000 7fc2dfed7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-06T13:57:40.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:39 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:57:39.928+0000 7fc2dfed7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-06T13:57:41.009 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:40 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:57:40.773+0000 7fc2dfed7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T13:57:41.009 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:41 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:57:41.004+0000 7fc2dfed7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T13:57:41.135 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:40 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:57:40.818+0000 7f5d14331200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-06T13:57:41.135 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:57:41.040+0000 7f5d14331200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-06T13:57:41.136 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:41 smithi077 ceph-mon[84247]: Standby manager daemon x restarted 2024-05-06T13:57:41.136 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:41 smithi077 ceph-mon[84247]: Standby manager daemon x started 2024-05-06T13:57:41.136 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:41 smithi077 ceph-mon[84247]: from='mgr.? 172.21.15.170:0/4267503566' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T13:57:41.136 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:41 smithi077 ceph-mon[84247]: from='mgr.? 172.21.15.170:0/4267503566' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:57:41.136 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:41 smithi077 ceph-mon[84247]: from='mgr.? 172.21.15.170:0/4267503566' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T13:57:41.136 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:41 smithi077 ceph-mon[84247]: from='mgr.? 172.21.15.170:0/4267503566' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:57:41.136 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:41 smithi077 ceph-mon[84247]: Active manager daemon y restarted 2024-05-06T13:57:41.136 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:41 smithi077 ceph-mon[84247]: Activating manager daemon y 2024-05-06T13:57:41.137 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:41 smithi077 ceph-mon[84247]: osdmap e113: 8 total, 8 up, 8 in 2024-05-06T13:57:41.137 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:41 smithi077 ceph-mon[84247]: mgrmap e45: y(active, starting, since 0.00942048s), standbys: x 2024-05-06T13:57:41.137 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:41 smithi077 ceph-mon[27166]: Standby manager daemon x restarted 2024-05-06T13:57:41.137 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:41 smithi077 ceph-mon[27166]: Standby manager daemon x started 2024-05-06T13:57:41.137 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:41 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/4267503566' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T13:57:41.137 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:41 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/4267503566' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:57:41.137 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:41 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/4267503566' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T13:57:41.137 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:41 smithi077 ceph-mon[27166]: from='mgr.? 172.21.15.170:0/4267503566' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:57:41.137 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:41 smithi077 ceph-mon[27166]: Active manager daemon y restarted 2024-05-06T13:57:41.137 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:41 smithi077 ceph-mon[27166]: Activating manager daemon y 2024-05-06T13:57:41.138 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:41 smithi077 ceph-mon[27166]: osdmap e113: 8 total, 8 up, 8 in 2024-05-06T13:57:41.138 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:41 smithi077 ceph-mon[27166]: mgrmap e45: y(active, starting, since 0.00942048s), standbys: x 2024-05-06T13:57:41.326 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:41 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: [06/May/2024:13:57:41] ENGINE Bus STARTING 2024-05-06T13:57:41.327 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:41 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: CherryPy Checker: 2024-05-06T13:57:41.327 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:41 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: The Application mounted at '' has an empty config. 2024-05-06T13:57:41.327 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:41 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: 2024-05-06T13:57:41.327 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:41 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: [06/May/2024:13:57:41] ENGINE Serving on http://:::9283 2024-05-06T13:57:41.327 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 13:57:41 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x[58762]: [06/May/2024:13:57:41] ENGINE Bus STARTED 2024-05-06T13:57:41.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:41 smithi170 ceph-mon[65866]: Standby manager daemon x restarted 2024-05-06T13:57:41.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:41 smithi170 ceph-mon[65866]: Standby manager daemon x started 2024-05-06T13:57:41.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:41 smithi170 ceph-mon[65866]: from='mgr.? 172.21.15.170:0/4267503566' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-06T13:57:41.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:41 smithi170 ceph-mon[65866]: from='mgr.? 172.21.15.170:0/4267503566' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-06T13:57:41.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:41 smithi170 ceph-mon[65866]: from='mgr.? 172.21.15.170:0/4267503566' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-06T13:57:41.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:41 smithi170 ceph-mon[65866]: from='mgr.? 172.21.15.170:0/4267503566' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-06T13:57:41.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:41 smithi170 ceph-mon[65866]: Active manager daemon y restarted 2024-05-06T13:57:41.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:41 smithi170 ceph-mon[65866]: Activating manager daemon y 2024-05-06T13:57:41.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:41 smithi170 ceph-mon[65866]: osdmap e113: 8 total, 8 up, 8 in 2024-05-06T13:57:41.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:41 smithi170 ceph-mon[65866]: mgrmap e45: y(active, starting, since 0.00942048s), standbys: x 2024-05-06T13:57:41.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:57:41] ENGINE Bus STARTING 2024-05-06T13:57:41.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: CherryPy Checker: 2024-05-06T13:57:41.552 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: The Application mounted at '' has an empty config. 2024-05-06T13:57:41.553 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:57:41.553 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:57:41] ENGINE Serving on http://:::9283 2024-05-06T13:57:41.553 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:41 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:57:41] ENGINE Bus STARTED 2024-05-06T13:57:42.314 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:57:42.073+0000 7f5cd1795700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-05-06T13:57:42.314 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:57:42.314 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:57:42.314 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:57:42.314 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T13:57:42.314 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T13:57:42.315 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:57:42.315 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:57:42.315 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:57:42.315 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:57:42.315 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:57:42.315 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:57:42.315 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:57:42.315 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:57:42.315 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T13:57:42.315 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T13:57:42.316 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T13:57:42.316 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[84247]: Manager daemon y is now available 2024-05-06T13:57:42.316 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:57:42.316 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:57:42.316 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[84247]: from='mgr.25849 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:57:42.316 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:57:42.316 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[84247]: from='mgr.25849 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:57:42.316 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:57:42.316 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:57:42.317 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:57:42.317 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T13:57:42.317 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T13:57:42.317 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:57:42.317 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:57:42.317 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:57:42.317 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:57:42.317 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:57:42.317 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:57:42.317 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:57:42.317 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:57:42.318 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T13:57:42.318 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T13:57:42.318 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T13:57:42.318 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[27166]: Manager daemon y is now available 2024-05-06T13:57:42.318 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:57:42.318 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:57:42.318 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[27166]: from='mgr.25849 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:57:42.318 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:57:42.318 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:42 smithi077 ceph-mon[27166]: from='mgr.25849 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:57:42.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:42 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:57:42.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:42 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:57:42.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:42 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:57:42.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:42 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-06T13:57:42.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:42 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-06T13:57:42.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:42 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:57:42.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:42 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T13:57:42.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:42 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T13:57:42.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:42 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T13:57:42.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:42 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T13:57:42.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:42 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T13:57:42.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:42 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T13:57:42.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:42 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T13:57:42.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:42 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-06T13:57:42.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:42 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-06T13:57:42.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:42 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-06T13:57:42.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:42 smithi170 ceph-mon[65866]: Manager daemon y is now available 2024-05-06T13:57:42.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:42 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:57:42.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:42 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:57:42.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:42 smithi170 ceph-mon[65866]: from='mgr.25849 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-06T13:57:42.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:42 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:57:42.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:42 smithi170 ceph-mon[65866]: from='mgr.25849 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-06T13:57:43.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:43 smithi077 ceph-mon[84247]: [06/May/2024:13:57:41] ENGINE Bus STARTING 2024-05-06T13:57:43.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:43 smithi077 ceph-mon[84247]: [06/May/2024:13:57:41] ENGINE Serving on https://172.21.15.77:7150 2024-05-06T13:57:43.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:43 smithi077 ceph-mon[84247]: [06/May/2024:13:57:41] ENGINE Serving on http://172.21.15.77:8765 2024-05-06T13:57:43.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:43 smithi077 ceph-mon[84247]: [06/May/2024:13:57:41] ENGINE Bus STARTED 2024-05-06T13:57:43.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:43 smithi077 ceph-mon[84247]: mgrmap e46: y(active, since 1.02959s), standbys: x 2024-05-06T13:57:43.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:43 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:57:43.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:43 smithi077 ceph-mon[84247]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:57:43.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:43 smithi077 ceph-mon[27166]: [06/May/2024:13:57:41] ENGINE Bus STARTING 2024-05-06T13:57:43.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:43 smithi077 ceph-mon[27166]: [06/May/2024:13:57:41] ENGINE Serving on https://172.21.15.77:7150 2024-05-06T13:57:43.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:43 smithi077 ceph-mon[27166]: [06/May/2024:13:57:41] ENGINE Serving on http://172.21.15.77:8765 2024-05-06T13:57:43.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:43 smithi077 ceph-mon[27166]: [06/May/2024:13:57:41] ENGINE Bus STARTED 2024-05-06T13:57:43.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:43 smithi077 ceph-mon[27166]: mgrmap e46: y(active, since 1.02959s), standbys: x 2024-05-06T13:57:43.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:43 smithi077 ceph-mon[27166]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:57:43.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:43 smithi077 ceph-mon[27166]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:57:43.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:43 smithi170 ceph-mon[65866]: [06/May/2024:13:57:41] ENGINE Bus STARTING 2024-05-06T13:57:43.387 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:43 smithi170 ceph-mon[65866]: [06/May/2024:13:57:41] ENGINE Serving on https://172.21.15.77:7150 2024-05-06T13:57:43.387 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:43 smithi170 ceph-mon[65866]: [06/May/2024:13:57:41] ENGINE Serving on http://172.21.15.77:8765 2024-05-06T13:57:43.387 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:43 smithi170 ceph-mon[65866]: [06/May/2024:13:57:41] ENGINE Bus STARTED 2024-05-06T13:57:43.387 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:43 smithi170 ceph-mon[65866]: mgrmap e46: y(active, since 1.02959s), standbys: x 2024-05-06T13:57:43.387 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:43 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:57:43.387 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:43 smithi170 ceph-mon[65866]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:57:44.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:57:43] "GET /metrics HTTP/1.1" 200 35203 "" "Prometheus/2.43.0" 2024-05-06T13:57:44.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:44 smithi170 ceph-mon[65866]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:57:44.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:44 smithi170 ceph-mon[65866]: mgrmap e47: y(active, since 2s), standbys: x 2024-05-06T13:57:44.409 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:44 smithi077 ceph-mon[84247]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:57:44.409 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:44 smithi077 ceph-mon[84247]: mgrmap e47: y(active, since 2s), standbys: x 2024-05-06T13:57:44.410 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:44 smithi077 ceph-mon[27166]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:57:44.410 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:44 smithi077 ceph-mon[27166]: mgrmap e47: y(active, since 2s), standbys: x 2024-05-06T13:57:46.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:46 smithi077 ceph-mon[84247]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:57:46.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:46 smithi077 ceph-mon[84247]: mgrmap e48: y(active, since 4s), standbys: x 2024-05-06T13:57:46.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:46 smithi077 ceph-mon[84247]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:46.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:46 smithi077 ceph-mon[84247]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:46.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:46 smithi077 ceph-mon[84247]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:46.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:46 smithi077 ceph-mon[84247]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:46.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:46 smithi077 ceph-mon[27166]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:57:46.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:46 smithi077 ceph-mon[27166]: mgrmap e48: y(active, since 4s), standbys: x 2024-05-06T13:57:46.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:46 smithi077 ceph-mon[27166]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:46.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:46 smithi077 ceph-mon[27166]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:46.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:46 smithi077 ceph-mon[27166]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:46.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:46 smithi077 ceph-mon[27166]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:46.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:46 smithi170 ceph-mon[65866]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:57:46.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:46 smithi170 ceph-mon[65866]: mgrmap e48: y(active, since 4s), standbys: x 2024-05-06T13:57:46.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:46 smithi170 ceph-mon[65866]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:46.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:46 smithi170 ceph-mon[65866]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:46.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:46 smithi170 ceph-mon[65866]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:46.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:46 smithi170 ceph-mon[65866]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:47.889 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:47 smithi077 ceph-mon[84247]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:47.890 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:47 smithi077 ceph-mon[84247]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:47.890 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:47 smithi077 ceph-mon[84247]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:47.890 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:47 smithi077 ceph-mon[84247]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:47.890 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:47 smithi077 ceph-mon[84247]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:57:47.890 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:47 smithi077 ceph-mon[27166]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:47.890 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:47 smithi077 ceph-mon[27166]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:47.890 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:47 smithi077 ceph-mon[27166]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:47.890 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:47 smithi077 ceph-mon[27166]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:47.890 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:47 smithi077 ceph-mon[27166]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:57:48.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:47 smithi170 ceph-mon[65866]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:48.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:47 smithi170 ceph-mon[65866]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:48.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:47 smithi170 ceph-mon[65866]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:48.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:47 smithi170 ceph-mon[65866]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:48.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:47 smithi170 ceph-mon[65866]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-05-06T13:57:50.236 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:50 smithi077 ceph-mon[84247]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-05-06T13:57:50.236 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:50 smithi077 ceph-mon[27166]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-05-06T13:57:50.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:50 smithi170 ceph-mon[65866]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-05-06T13:57:51.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:51 smithi077 ceph-mon[27166]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:51.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:51 smithi077 ceph-mon[27166]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:51.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:51 smithi077 ceph-mon[27166]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:51.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:51 smithi077 ceph-mon[27166]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:51.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:51 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:57:51.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:51 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:57:51.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:51 smithi077 ceph-mon[84247]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:51.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:51 smithi077 ceph-mon[84247]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:51.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:51 smithi077 ceph-mon[84247]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:51.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:51 smithi077 ceph-mon[84247]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:51.555 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:51 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:57:51.555 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:51 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:57:51.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:51 smithi170 ceph-mon[65866]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:51.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:51 smithi170 ceph-mon[65866]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:51.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:51 smithi170 ceph-mon[65866]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:51.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:51 smithi170 ceph-mon[65866]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:51.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:51 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:57:51.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:51 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:57:52.229 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[84247]: Detected new or changed devices on smithi077 2024-05-06T13:57:52.229 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[84247]: Updating smithi077:/etc/ceph/ceph.conf 2024-05-06T13:57:52.229 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[84247]: Updating smithi170:/etc/ceph/ceph.conf 2024-05-06T13:57:52.229 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[84247]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:57:52.229 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[84247]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:57:52.229 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[84247]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:57:52.230 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[84247]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:57:52.230 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[84247]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:57:52.230 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[84247]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:57:52.230 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[84247]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-06T13:57:52.230 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[84247]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:52.230 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[84247]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:52.230 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[84247]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:52.230 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[84247]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:52.230 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[84247]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:52.230 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:57:52.230 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:57:52.230 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-05-06T13:57:52.230 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-05-06T13:57:52.231 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[27166]: Detected new or changed devices on smithi077 2024-05-06T13:57:52.231 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[27166]: Updating smithi077:/etc/ceph/ceph.conf 2024-05-06T13:57:52.231 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[27166]: Updating smithi170:/etc/ceph/ceph.conf 2024-05-06T13:57:52.231 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[27166]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:57:52.231 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[27166]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:57:52.231 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[27166]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:57:52.231 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[27166]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:57:52.231 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[27166]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:57:52.231 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[27166]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:57:52.231 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[27166]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-06T13:57:52.231 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[27166]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:52.231 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[27166]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:52.231 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[27166]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:52.231 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[27166]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:52.232 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[27166]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:52.232 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:57:52.232 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:57:52.232 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-05-06T13:57:52.232 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:52 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-05-06T13:57:52.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:52 smithi170 ceph-mon[65866]: Detected new or changed devices on smithi077 2024-05-06T13:57:52.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:52 smithi170 ceph-mon[65866]: Updating smithi077:/etc/ceph/ceph.conf 2024-05-06T13:57:52.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:52 smithi170 ceph-mon[65866]: Updating smithi170:/etc/ceph/ceph.conf 2024-05-06T13:57:52.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:52 smithi170 ceph-mon[65866]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:57:52.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:52 smithi170 ceph-mon[65866]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-06T13:57:52.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:52 smithi170 ceph-mon[65866]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:57:52.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:52 smithi170 ceph-mon[65866]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-05-06T13:57:52.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:52 smithi170 ceph-mon[65866]: Updating smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:57:52.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:52 smithi170 ceph-mon[65866]: Updating smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-06T13:57:52.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:52 smithi170 ceph-mon[65866]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-06T13:57:52.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:52 smithi170 ceph-mon[65866]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:52.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:52 smithi170 ceph-mon[65866]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:52.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:52 smithi170 ceph-mon[65866]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:52.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:52 smithi170 ceph-mon[65866]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:52.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:52 smithi170 ceph-mon[65866]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:52.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:52 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:57:52.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:52 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:57:52.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:52 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-05-06T13:57:52.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:52 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-05-06T13:57:53.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:53 smithi077 ceph-mon[84247]: Upgrade: It appears safe to stop mon.c 2024-05-06T13:57:53.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:53 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:57:53.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:53 smithi077 ceph-mon[84247]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:53.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:53 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:57:53.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:53 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T13:57:53.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:53 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:57:53.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:53 smithi077 ceph-mon[27166]: Upgrade: It appears safe to stop mon.c 2024-05-06T13:57:53.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:53 smithi077 ceph-mon[27166]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:57:53.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:53 smithi077 ceph-mon[27166]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:53.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:53 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:57:53.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:53 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T13:57:53.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:53 smithi077 ceph-mon[27166]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:57:53.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:53 smithi170 ceph-mon[65866]: Upgrade: It appears safe to stop mon.c 2024-05-06T13:57:53.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:53 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:57:53.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:53 smithi170 ceph-mon[65866]: from='mgr.25849 ' entity='mgr.y' 2024-05-06T13:57:53.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:53 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:57:53.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:53 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T13:57:53.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:53 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:57:53.802 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:57:53] "GET /metrics HTTP/1.1" 200 35203 "" "Prometheus/2.43.0" 2024-05-06T13:57:54.487 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:54 smithi077 ceph-mon[84247]: Upgrade: Updating mon.c 2024-05-06T13:57:54.487 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:54 smithi077 ceph-mon[84247]: Deploying daemon mon.c on smithi077 2024-05-06T13:57:54.488 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:54 smithi077 ceph-mon[84247]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-06T13:57:54.488 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:54 smithi077 ceph-mon[27166]: Upgrade: Updating mon.c 2024-05-06T13:57:54.488 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:54 smithi077 ceph-mon[27166]: Deploying daemon mon.c on smithi077 2024-05-06T13:57:54.488 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:54 smithi077 ceph-mon[27166]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-06T13:57:54.581 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:54 smithi170 ceph-mon[65866]: Upgrade: Updating mon.c 2024-05-06T13:57:54.581 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:54 smithi170 ceph-mon[65866]: Deploying daemon mon.c on smithi077 2024-05-06T13:57:54.581 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:54 smithi170 ceph-mon[65866]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-06T13:57:55.084 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:54 smithi077 systemd[1]: Stopping Ceph mon.c for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:57:55.542 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:55 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-c[27162]: 2024-05-06T13:57:55.082+0000 7f3f795c9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T13:57:55.543 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:55 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-c[27162]: 2024-05-06T13:57:55.082+0000 7f3f795c9700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-05-06T13:57:55.543 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:55 smithi077 podman[89035]: 2024-05-06 13:57:55.25669532 +0000 UTC m=+0.294097392 container died 83af336ef762e9b1a830dbf32bb4ece5ad1324f6f1719d4fd97d5f1cac146890 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-c, name=centos-stream, maintainer=Guillaume Abrioux , architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, com.redhat.component=centos-stream-container, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, GIT_CLEAN=True, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8) 2024-05-06T13:57:55.543 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:55 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:57:55] ENGINE Bus STOPPING 2024-05-06T13:57:55.543 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:55 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:57:55] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-06T13:57:55.543 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:55 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:57:55] ENGINE Bus STOPPED 2024-05-06T13:57:55.543 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:55 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:57:55] ENGINE Bus STARTING 2024-05-06T13:57:55.543 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:55 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:57:55] ENGINE Serving on http://:::9283 2024-05-06T13:57:55.543 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:57:55 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:13:57:55] ENGINE Bus STARTED 2024-05-06T13:57:55.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:55 smithi077 podman[89035]: 2024-05-06 13:57:55.642018132 +0000 UTC m=+0.679420211 container cleanup 83af336ef762e9b1a830dbf32bb4ece5ad1324f6f1719d4fd97d5f1cac146890 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-c, io.buildah.version=1.19.8, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, io.openshift.expose-services=, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, release=754, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, ceph=True, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-05-06T13:57:55.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:55 smithi077 bash[89035]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-c 2024-05-06T13:57:56.255 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:55 smithi077 podman[89060]: 2024-05-06 13:57:55.901834026 +0000 UTC m=+0.638579840 container remove 83af336ef762e9b1a830dbf32bb4ece5ad1324f6f1719d4fd97d5f1cac146890 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-c, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, io.openshift.expose-services=, io.buildah.version=1.19.8, ceph=True, vcs-type=git, name=centos-stream, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, distribution-scope=public, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream) 2024-05-06T13:57:56.255 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mon.c.service: Deactivated successfully. 2024-05-06T13:57:56.255 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 systemd[1]: Stopped Ceph mon.c for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:57:56.256 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mon.c.service: Consumed 15.457s CPU time. 2024-05-06T13:57:56.256 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 systemd[1]: Starting Ceph mon.c for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:57:56.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 podman[89193]: 2024-05-06 13:57:56.374885974 +0000 UTC m=+0.018160801 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:57:56.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 podman[89193]: 2024-05-06 13:57:56.497439986 +0000 UTC m=+0.140714865 container create 8a473cc2ffd10a1f9daa349f5b49d6e9dc2fac143a6c9c959e3e8505e0d0fddd (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430) 2024-05-06T13:57:57.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 podman[89193]: 2024-05-06 13:57:56.758032462 +0000 UTC m=+0.401307293 container init 8a473cc2ffd10a1f9daa349f5b49d6e9dc2fac143a6c9c959e3e8505e0d0fddd (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0) 2024-05-06T13:57:57.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 podman[89193]: 2024-05-06 13:57:56.761152074 +0000 UTC m=+0.404426895 container start 8a473cc2ffd10a1f9daa349f5b49d6e9dc2fac143a6c9c959e3e8505e0d0fddd (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-c, RELEASE=squid-70ce953, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-05-06T13:57:57.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: set uid:gid to 167:167 (ceph:ceph) 2024-05-06T13:57:57.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev), process ceph-mon, pid 2 2024-05-06T13:57:57.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: pidfile_write: ignore empty --pid-file 2024-05-06T13:57:57.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: load: jerasure load: lrc 2024-05-06T13:57:57.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: RocksDB version: 7.9.2 2024-05-06T13:57:57.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Git sha 0 2024-05-06T13:57:57.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Compile date 2024-05-03 15:08:47 2024-05-06T13:57:57.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: DB SUMMARY 2024-05-06T13:57:57.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: DB Session ID: XQ8O6UMXGEKMGUY1ZQY8 2024-05-06T13:57:57.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: CURRENT file: CURRENT 2024-05-06T13:57:57.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: IDENTITY file: IDENTITY 2024-05-06T13:57:57.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: MANIFEST file: MANIFEST-000009 size: 3203 Bytes 2024-05-06T13:57:57.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000060.sst 2024-05-06T13:57:57.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000058.log size: 2364200 ; 2024-05-06T13:57:57.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.error_if_exists: 0 2024-05-06T13:57:57.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.create_if_missing: 0 2024-05-06T13:57:57.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.paranoid_checks: 1 2024-05-06T13:57:57.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-06T13:57:57.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-06T13:57:57.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-06T13:57:57.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.env: 0x55ddda7d8c60 2024-05-06T13:57:57.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.fs: PosixFileSystem 2024-05-06T13:57:57.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.info_log: 0x55dddc8c3ac0 2024-05-06T13:57:57.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.max_file_opening_threads: 16 2024-05-06T13:57:57.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.statistics: (nil) 2024-05-06T13:57:57.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.use_fsync: 0 2024-05-06T13:57:57.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.max_log_file_size: 0 2024-05-06T13:57:57.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-06T13:57:57.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-06T13:57:57.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.keep_log_file_num: 1000 2024-05-06T13:57:57.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.recycle_log_file_num: 0 2024-05-06T13:57:57.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.allow_fallocate: 1 2024-05-06T13:57:57.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.allow_mmap_reads: 0 2024-05-06T13:57:57.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.allow_mmap_writes: 0 2024-05-06T13:57:57.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.use_direct_reads: 0 2024-05-06T13:57:57.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-06T13:57:57.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.create_missing_column_families: 0 2024-05-06T13:57:57.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.db_log_dir: 2024-05-06T13:57:57.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.wal_dir: 2024-05-06T13:57:57.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-06T13:57:57.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-06T13:57:57.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-06T13:57:57.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-06T13:57:57.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-06T13:57:57.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-06T13:57:57.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.advise_random_on_open: 1 2024-05-06T13:57:57.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.db_write_buffer_size: 0 2024-05-06T13:57:57.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.write_buffer_manager: 0x55dddc896820 2024-05-06T13:57:57.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-06T13:57:57.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-06T13:57:57.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-06T13:57:57.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.rate_limiter: (nil) 2024-05-06T13:57:57.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-06T13:57:57.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.wal_recovery_mode: 2 2024-05-06T13:57:57.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.enable_thread_tracking: 0 2024-05-06T13:57:57.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.enable_pipelined_write: 0 2024-05-06T13:57:57.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.unordered_write: 0 2024-05-06T13:57:57.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-06T13:57:57.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-06T13:57:57.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-06T13:57:57.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-06T13:57:57.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.row_cache: None 2024-05-06T13:57:57.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.wal_filter: None 2024-05-06T13:57:57.057 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-06T13:57:57.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.allow_ingest_behind: 0 2024-05-06T13:57:57.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.two_write_queues: 0 2024-05-06T13:57:57.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.manual_wal_flush: 0 2024-05-06T13:57:57.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.wal_compression: 0 2024-05-06T13:57:57.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.atomic_flush: 0 2024-05-06T13:57:57.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-06T13:57:57.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-06T13:57:57.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-06T13:57:57.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.log_readahead_size: 0 2024-05-06T13:57:57.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-06T13:57:57.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.best_efforts_recovery: 0 2024-05-06T13:57:57.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-06T13:57:57.059 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-06T13:57:57.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.allow_data_in_errors: 0 2024-05-06T13:57:57.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.db_host_id: __hostname__ 2024-05-06T13:57:57.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-06T13:57:57.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.max_background_jobs: 2 2024-05-06T13:57:57.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.max_background_compactions: -1 2024-05-06T13:57:57.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.max_subcompactions: 1 2024-05-06T13:57:57.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-06T13:57:57.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-06T13:57:57.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-06T13:57:57.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.max_total_wal_size: 0 2024-05-06T13:57:57.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-06T13:57:57.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-06T13:57:57.060 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-06T13:57:57.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-06T13:57:57.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.max_open_files: -1 2024-05-06T13:57:57.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.bytes_per_sync: 0 2024-05-06T13:57:57.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-06T13:57:57.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-06T13:57:57.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.compaction_readahead_size: 0 2024-05-06T13:57:57.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.max_background_flushes: -1 2024-05-06T13:57:57.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Compression algorithms supported: 2024-05-06T13:57:57.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-06T13:57:57.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: kZSTD supported: 0 2024-05-06T13:57:57.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: kXpressCompression supported: 0 2024-05-06T13:57:57.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: kLZ4HCCompression supported: 1 2024-05-06T13:57:57.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: kZlibCompression supported: 1 2024-05-06T13:57:57.061 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: kSnappyCompression supported: 1 2024-05-06T13:57:57.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: kLZ4Compression supported: 1 2024-05-06T13:57:57.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: kBZip2Compression supported: 0 2024-05-06T13:57:57.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-06T13:57:57.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-06T13:57:57.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-05-06T13:57:57.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-06T13:57:57.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-06T13:57:57.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.merge_operator: 2024-05-06T13:57:57.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.compaction_filter: None 2024-05-06T13:57:57.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.compaction_filter_factory: None 2024-05-06T13:57:57.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.sst_partitioner_factory: None 2024-05-06T13:57:57.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-06T13:57:57.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-06T13:57:57.062 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55dddc8c3bc0) 2024-05-06T13:57:57.063 INFO:journalctl@ceph.mon.c.smithi077.stdout: cache_index_and_filter_blocks: 1 2024-05-06T13:57:57.063 INFO:journalctl@ceph.mon.c.smithi077.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-06T13:57:57.063 INFO:journalctl@ceph.mon.c.smithi077.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-06T13:57:57.063 INFO:journalctl@ceph.mon.c.smithi077.stdout: pin_top_level_index_and_filter: 1 2024-05-06T13:57:57.063 INFO:journalctl@ceph.mon.c.smithi077.stdout: index_type: 0 2024-05-06T13:57:57.063 INFO:journalctl@ceph.mon.c.smithi077.stdout: data_block_index_type: 0 2024-05-06T13:57:57.063 INFO:journalctl@ceph.mon.c.smithi077.stdout: index_shortening: 1 2024-05-06T13:57:57.063 INFO:journalctl@ceph.mon.c.smithi077.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-06T13:57:57.063 INFO:journalctl@ceph.mon.c.smithi077.stdout: checksum: 4 2024-05-06T13:57:57.063 INFO:journalctl@ceph.mon.c.smithi077.stdout: no_block_cache: 0 2024-05-06T13:57:57.063 INFO:journalctl@ceph.mon.c.smithi077.stdout: block_cache: 0x55dddc8c7090 2024-05-06T13:57:57.063 INFO:journalctl@ceph.mon.c.smithi077.stdout: block_cache_name: BinnedLRUCache 2024-05-06T13:57:57.063 INFO:journalctl@ceph.mon.c.smithi077.stdout: block_cache_options: 2024-05-06T13:57:57.063 INFO:journalctl@ceph.mon.c.smithi077.stdout: capacity : 536870912 2024-05-06T13:57:57.063 INFO:journalctl@ceph.mon.c.smithi077.stdout: num_shard_bits : 4 2024-05-06T13:57:57.063 INFO:journalctl@ceph.mon.c.smithi077.stdout: strict_capacity_limit : 0 2024-05-06T13:57:57.064 INFO:journalctl@ceph.mon.c.smithi077.stdout: high_pri_pool_ratio: 0.000 2024-05-06T13:57:57.064 INFO:journalctl@ceph.mon.c.smithi077.stdout: block_cache_compressed: (nil) 2024-05-06T13:57:57.064 INFO:journalctl@ceph.mon.c.smithi077.stdout: persistent_cache: (nil) 2024-05-06T13:57:57.064 INFO:journalctl@ceph.mon.c.smithi077.stdout: block_size: 4096 2024-05-06T13:57:57.064 INFO:journalctl@ceph.mon.c.smithi077.stdout: block_size_deviation: 10 2024-05-06T13:57:57.064 INFO:journalctl@ceph.mon.c.smithi077.stdout: block_restart_interval: 16 2024-05-06T13:57:57.064 INFO:journalctl@ceph.mon.c.smithi077.stdout: index_block_restart_interval: 1 2024-05-06T13:57:57.064 INFO:journalctl@ceph.mon.c.smithi077.stdout: metadata_block_size: 4096 2024-05-06T13:57:57.064 INFO:journalctl@ceph.mon.c.smithi077.stdout: partition_filters: 0 2024-05-06T13:57:57.064 INFO:journalctl@ceph.mon.c.smithi077.stdout: use_delta_encoding: 1 2024-05-06T13:57:57.064 INFO:journalctl@ceph.mon.c.smithi077.stdout: filter_policy: bloomfilter 2024-05-06T13:57:57.064 INFO:journalctl@ceph.mon.c.smithi077.stdout: whole_key_filtering: 1 2024-05-06T13:57:57.064 INFO:journalctl@ceph.mon.c.smithi077.stdout: verify_compression: 0 2024-05-06T13:57:57.064 INFO:journalctl@ceph.mon.c.smithi077.stdout: read_amp_bytes_per_bit: 0 2024-05-06T13:57:57.065 INFO:journalctl@ceph.mon.c.smithi077.stdout: format_version: 5 2024-05-06T13:57:57.065 INFO:journalctl@ceph.mon.c.smithi077.stdout: enable_index_compression: 1 2024-05-06T13:57:57.065 INFO:journalctl@ceph.mon.c.smithi077.stdout: block_align: 0 2024-05-06T13:57:57.065 INFO:journalctl@ceph.mon.c.smithi077.stdout: max_auto_readahead_size: 262144 2024-05-06T13:57:57.065 INFO:journalctl@ceph.mon.c.smithi077.stdout: prepopulate_block_cache: 0 2024-05-06T13:57:57.065 INFO:journalctl@ceph.mon.c.smithi077.stdout: initial_auto_readahead_size: 8192 2024-05-06T13:57:57.065 INFO:journalctl@ceph.mon.c.smithi077.stdout: num_file_reads_for_auto_readahead: 2 2024-05-06T13:57:57.065 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.write_buffer_size: 33554432 2024-05-06T13:57:57.065 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.max_write_buffer_number: 2 2024-05-06T13:57:57.065 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.compression: NoCompression 2024-05-06T13:57:57.065 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.bottommost_compression: Disabled 2024-05-06T13:57:57.065 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.prefix_extractor: nullptr 2024-05-06T13:57:57.066 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-06T13:57:57.066 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.num_levels: 7 2024-05-06T13:57:57.066 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-06T13:57:57.066 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-06T13:57:57.066 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-06T13:57:57.066 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-06T13:57:57.066 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-06T13:57:57.066 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-06T13:57:57.066 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-06T13:57:57.066 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-06T13:57:57.066 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-06T13:57:57.066 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-06T13:57:57.066 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-06T13:57:57.066 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-06T13:57:57.067 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-06T13:57:57.067 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.compression_opts.level: 32767 2024-05-06T13:57:57.067 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.compression_opts.strategy: 0 2024-05-06T13:57:57.067 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-06T13:57:57.067 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-06T13:57:57.067 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-06T13:57:57.067 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-06T13:57:57.067 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.compression_opts.enabled: false 2024-05-06T13:57:57.067 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-06T13:57:57.067 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-06T13:57:57.067 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-06T13:57:57.067 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-06T13:57:57.067 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.target_file_size_base: 67108864 2024-05-06T13:57:57.067 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-06T13:57:57.067 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-06T13:57:57.068 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-06T13:57:57.068 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-06T13:57:57.068 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-06T13:57:57.068 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-06T13:57:57.068 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-06T13:57:57.068 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-06T13:57:57.068 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-06T13:57:57.068 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-06T13:57:57.068 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-06T13:57:57.068 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-06T13:57:57.068 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-06T13:57:57.068 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-06T13:57:57.068 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.arena_block_size: 1048576 2024-05-06T13:57:57.068 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-06T13:57:57.068 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-06T13:57:57.068 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.disable_auto_compactions: 0 2024-05-06T13:57:57.069 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-06T13:57:57.069 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-06T13:57:57.069 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-06T13:57:57.069 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-06T13:57:57.069 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-06T13:57:57.069 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-06T13:57:57.069 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-06T13:57:57.069 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-06T13:57:57.069 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-06T13:57:57.069 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-06T13:57:57.069 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.table_properties_collectors: 2024-05-06T13:57:57.069 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.inplace_update_support: 0 2024-05-06T13:57:57.069 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-06T13:57:57.069 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-06T13:57:57.070 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-06T13:57:57.070 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-06T13:57:57.070 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.bloom_locality: 0 2024-05-06T13:57:57.070 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.max_successive_merges: 0 2024-05-06T13:57:57.070 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-06T13:57:57.070 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.paranoid_file_checks: 0 2024-05-06T13:57:57.070 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.force_consistency_checks: 1 2024-05-06T13:57:57.070 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.report_bg_io_stats: 0 2024-05-06T13:57:57.070 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.ttl: 2592000 2024-05-06T13:57:57.070 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-06T13:57:57.070 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-06T13:57:57.070 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-06T13:57:57.070 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.enable_blob_files: false 2024-05-06T13:57:57.070 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.min_blob_size: 0 2024-05-06T13:57:57.070 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.blob_file_size: 268435456 2024-05-06T13:57:57.071 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-06T13:57:57.071 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-06T13:57:57.071 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-06T13:57:57.071 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-06T13:57:57.071 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-06T13:57:57.071 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.blob_file_starting_level: 0 2024-05-06T13:57:57.071 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-06T13:57:57.071 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 60.sst 2024-05-06T13:57:57.071 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-05-06T13:57:57.071 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 62, last_sequence is 32424, log_number is 58,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-06T13:57:57.071 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 58 2024-05-06T13:57:57.071 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 1c965386-f64c-4736-a27d-31c3b9e5d48e 2024-05-06T13:57:57.071 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715003876790342, "job": 1, "event": "recovery_started", "wal_files": [58]} 2024-05-06T13:57:57.071 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #58 mode 2 2024-05-06T13:57:57.071 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715003876796260, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 63, "file_size": 1494612, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 32430, "largest_seqno": 32827, "table_properties": {"data_size": 1491704, "index_size": 1382, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 517, "raw_key_size": 4667, "raw_average_key_size": 24, "raw_value_size": 1487267, "raw_average_value_size": 7706, "num_data_blocks": 62, "num_entries": 193, "num_filter_entries": 193, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715003876, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "1c965386-f64c-4736-a27d-31c3b9e5d48e", "db_session_id": "XQ8O6UMXGEKMGUY1ZQY8", "orig_file_number": 63, "seqno_to_time_mapping": "N/A"}} 2024-05-06T13:57:57.071 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715003876796360, "job": 1, "event": "recovery_finished"} 2024-05-06T13:57:57.072 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: [db/version_set.cc:5047] Creating manifest 65 2024-05-06T13:57:57.072 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-05-06T13:57:57.072 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000058.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-06T13:57:57.072 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55dddc9b8000 2024-05-06T13:57:57.072 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: DB pointer 0x55dddc9a2000 2024-05-06T13:57:57.072 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-06T13:57:57.072 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-06T13:57:57.072 INFO:journalctl@ceph.mon.c.smithi077.stdout: ** DB Stats ** 2024-05-06T13:57:57.072 INFO:journalctl@ceph.mon.c.smithi077.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T13:57:57.072 INFO:journalctl@ceph.mon.c.smithi077.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-06T13:57:57.072 INFO:journalctl@ceph.mon.c.smithi077.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-06T13:57:57.072 INFO:journalctl@ceph.mon.c.smithi077.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-06T13:57:57.072 INFO:journalctl@ceph.mon.c.smithi077.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-06T13:57:57.072 INFO:journalctl@ceph.mon.c.smithi077.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-06T13:57:57.072 INFO:journalctl@ceph.mon.c.smithi077.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-06T13:57:57.073 INFO:journalctl@ceph.mon.c.smithi077.stdout: 2024-05-06T13:57:57.073 INFO:journalctl@ceph.mon.c.smithi077.stdout: ** Compaction Stats [default] ** 2024-05-06T13:57:57.073 INFO:journalctl@ceph.mon.c.smithi077.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-06T13:57:57.073 INFO:journalctl@ceph.mon.c.smithi077.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-06T13:57:57.073 INFO:journalctl@ceph.mon.c.smithi077.stdout: L0 1/0 1.43 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 516.3 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-05-06T13:57:57.073 INFO:journalctl@ceph.mon.c.smithi077.stdout: L6 1/0 10.04 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-05-06T13:57:57.073 INFO:journalctl@ceph.mon.c.smithi077.stdout: Sum 2/0 11.46 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 516.3 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-05-06T13:57:57.073 INFO:journalctl@ceph.mon.c.smithi077.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 516.3 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-05-06T13:57:57.073 INFO:journalctl@ceph.mon.c.smithi077.stdout: 2024-05-06T13:57:57.073 INFO:journalctl@ceph.mon.c.smithi077.stdout: ** Compaction Stats [default] ** 2024-05-06T13:57:57.073 INFO:journalctl@ceph.mon.c.smithi077.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-06T13:57:57.073 INFO:journalctl@ceph.mon.c.smithi077.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-06T13:57:57.073 INFO:journalctl@ceph.mon.c.smithi077.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 516.3 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-05-06T13:57:57.073 INFO:journalctl@ceph.mon.c.smithi077.stdout: 2024-05-06T13:57:57.073 INFO:journalctl@ceph.mon.c.smithi077.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-06T13:57:57.074 INFO:journalctl@ceph.mon.c.smithi077.stdout: 2024-05-06T13:57:57.074 INFO:journalctl@ceph.mon.c.smithi077.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-06T13:57:57.074 INFO:journalctl@ceph.mon.c.smithi077.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-05-06T13:57:57.074 INFO:journalctl@ceph.mon.c.smithi077.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-06T13:57:57.074 INFO:journalctl@ceph.mon.c.smithi077.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-06T13:57:57.074 INFO:journalctl@ceph.mon.c.smithi077.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-06T13:57:57.074 INFO:journalctl@ceph.mon.c.smithi077.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-06T13:57:57.074 INFO:journalctl@ceph.mon.c.smithi077.stdout: Cumulative compaction: 0.00 GB write, 172.23 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T13:57:57.074 INFO:journalctl@ceph.mon.c.smithi077.stdout: Interval compaction: 0.00 GB write, 172.23 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-06T13:57:57.074 INFO:journalctl@ceph.mon.c.smithi077.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-06T13:57:57.074 INFO:journalctl@ceph.mon.c.smithi077.stdout: Block cache BinnedLRUCache@0x55dddc8c7090#2 capacity: 512.00 MB usage: 2.05 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-05-06T13:57:57.074 INFO:journalctl@ceph.mon.c.smithi077.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.59 KB,0.000113249%) IndexBlock(1,1.45 KB,0.000277162%) Misc(1,0.00 KB,0%) 2024-05-06T13:57:57.074 INFO:journalctl@ceph.mon.c.smithi077.stdout: 2024-05-06T13:57:57.074 INFO:journalctl@ceph.mon.c.smithi077.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-06T13:57:57.074 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: starting mon.c rank 1 at public addrs [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] at bind addrs [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:57:57.075 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: mon.c@-1(???) e3 preinit fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:57:57.075 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: mon.c@-1(???).mds e1 new map 2024-05-06T13:57:57.075 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: mon.c@-1(???).mds e1 print_map 2024-05-06T13:57:57.075 INFO:journalctl@ceph.mon.c.smithi077.stdout: e1 2024-05-06T13:57:57.075 INFO:journalctl@ceph.mon.c.smithi077.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-06T13:57:57.075 INFO:journalctl@ceph.mon.c.smithi077.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-05-06T13:57:57.075 INFO:journalctl@ceph.mon.c.smithi077.stdout: legacy client fscid: -1 2024-05-06T13:57:57.075 INFO:journalctl@ceph.mon.c.smithi077.stdout: 2024-05-06T13:57:57.075 INFO:journalctl@ceph.mon.c.smithi077.stdout: No filesystems configured 2024-05-06T13:57:57.075 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: mon.c@-1(???).osd e113 crush map has features 3314933000854323200, adjusting msgr requires 2024-05-06T13:57:57.075 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: mon.c@-1(???).osd e113 crush map has features 432629239337189376, adjusting msgr requires 2024-05-06T13:57:57.075 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: mon.c@-1(???).osd e113 crush map has features 432629239337189376, adjusting msgr requires 2024-05-06T13:57:57.075 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: mon.c@-1(???).osd e113 crush map has features 432629239337189376, adjusting msgr requires 2024-05-06T13:57:57.075 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 ceph-mon[89212]: mon.c@-1(???).paxosservice(auth 1..28) refresh upgraded, format 0 -> 3 2024-05-06T13:57:57.076 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 bash[89193]: 8a473cc2ffd10a1f9daa349f5b49d6e9dc2fac143a6c9c959e3e8505e0d0fddd 2024-05-06T13:57:57.076 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:56 smithi077 systemd[1]: Started Ceph mon.c for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:57:59.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[84247]: osdmap e113: 8 total, 8 up, 8 in 2024-05-06T13:57:59.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[84247]: mgrmap e48: y(active, since 17s), standbys: x 2024-05-06T13:57:59.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[84247]: overall HEALTH_OK 2024-05-06T13:57:59.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[84247]: mon.b calling monitor election 2024-05-06T13:57:59.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[84247]: mon.c calling monitor election 2024-05-06T13:57:59.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[84247]: mon.a calling monitor election 2024-05-06T13:57:59.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:57:59.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:57:59.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[84247]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-06T13:57:59.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:57:59.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[84247]: monmap epoch 4 2024-05-06T13:57:59.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[84247]: fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:57:59.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[84247]: last_changed 2024-05-06T13:57:58.099356+0000 2024-05-06T13:57:59.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[84247]: created 2024-05-06T13:26:34.664804+0000 2024-05-06T13:57:59.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[84247]: min_mon_release 19 (squid) 2024-05-06T13:57:59.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[84247]: election_strategy: 1 2024-05-06T13:57:59.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[84247]: 0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2024-05-06T13:57:59.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[84247]: 1: [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] mon.c 2024-05-06T13:57:59.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[84247]: 2: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.b 2024-05-06T13:57:59.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[84247]: fsmap 2024-05-06T13:57:59.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[84247]: osdmap e113: 8 total, 8 up, 8 in 2024-05-06T13:57:59.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[84247]: mgrmap e48: y(active, since 17s), standbys: x 2024-05-06T13:57:59.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[84247]: overall HEALTH_OK 2024-05-06T13:57:59.305 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:57:59.305 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:57:59.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[89212]: osdmap e113: 8 total, 8 up, 8 in 2024-05-06T13:57:59.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[89212]: mgrmap e48: y(active, since 17s), standbys: x 2024-05-06T13:57:59.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[89212]: overall HEALTH_OK 2024-05-06T13:57:59.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[89212]: mon.b calling monitor election 2024-05-06T13:57:59.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[89212]: mon.c calling monitor election 2024-05-06T13:57:59.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[89212]: mon.a calling monitor election 2024-05-06T13:57:59.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:57:59.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:57:59.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[89212]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-06T13:57:59.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:57:59.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[89212]: monmap epoch 4 2024-05-06T13:57:59.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[89212]: fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:57:59.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[89212]: last_changed 2024-05-06T13:57:58.099356+0000 2024-05-06T13:57:59.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[89212]: created 2024-05-06T13:26:34.664804+0000 2024-05-06T13:57:59.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[89212]: min_mon_release 19 (squid) 2024-05-06T13:57:59.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[89212]: election_strategy: 1 2024-05-06T13:57:59.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[89212]: 0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2024-05-06T13:57:59.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[89212]: 1: [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] mon.c 2024-05-06T13:57:59.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[89212]: 2: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.b 2024-05-06T13:57:59.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[89212]: fsmap 2024-05-06T13:57:59.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[89212]: osdmap e113: 8 total, 8 up, 8 in 2024-05-06T13:57:59.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[89212]: mgrmap e48: y(active, since 17s), standbys: x 2024-05-06T13:57:59.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[89212]: overall HEALTH_OK 2024-05-06T13:57:59.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:57:59.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:57:59 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:57:59.512 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:59 smithi170 ceph-mon[65866]: osdmap e113: 8 total, 8 up, 8 in 2024-05-06T13:57:59.512 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:59 smithi170 ceph-mon[65866]: mgrmap e48: y(active, since 17s), standbys: x 2024-05-06T13:57:59.512 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:59 smithi170 ceph-mon[65866]: overall HEALTH_OK 2024-05-06T13:57:59.513 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:59 smithi170 ceph-mon[65866]: mon.b calling monitor election 2024-05-06T13:57:59.513 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:59 smithi170 ceph-mon[65866]: mon.c calling monitor election 2024-05-06T13:57:59.513 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:59 smithi170 ceph-mon[65866]: mon.a calling monitor election 2024-05-06T13:57:59.513 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:59 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-06T13:57:59.513 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:59 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-06T13:57:59.513 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:59 smithi170 ceph-mon[65866]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-06T13:57:59.513 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:59 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-06T13:57:59.513 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:59 smithi170 ceph-mon[65866]: monmap epoch 4 2024-05-06T13:57:59.513 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:59 smithi170 ceph-mon[65866]: fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 2024-05-06T13:57:59.513 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:59 smithi170 ceph-mon[65866]: last_changed 2024-05-06T13:57:58.099356+0000 2024-05-06T13:57:59.513 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:59 smithi170 ceph-mon[65866]: created 2024-05-06T13:26:34.664804+0000 2024-05-06T13:57:59.513 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:59 smithi170 ceph-mon[65866]: min_mon_release 19 (squid) 2024-05-06T13:57:59.513 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:59 smithi170 ceph-mon[65866]: election_strategy: 1 2024-05-06T13:57:59.513 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:59 smithi170 ceph-mon[65866]: 0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2024-05-06T13:57:59.513 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:59 smithi170 ceph-mon[65866]: 1: [v2:172.21.15.77:3301/0,v1:172.21.15.77:6790/0] mon.c 2024-05-06T13:57:59.514 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:59 smithi170 ceph-mon[65866]: 2: [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] mon.b 2024-05-06T13:57:59.514 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:59 smithi170 ceph-mon[65866]: fsmap 2024-05-06T13:57:59.514 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:59 smithi170 ceph-mon[65866]: osdmap e113: 8 total, 8 up, 8 in 2024-05-06T13:57:59.514 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:59 smithi170 ceph-mon[65866]: mgrmap e48: y(active, since 17s), standbys: x 2024-05-06T13:57:59.514 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:59 smithi170 ceph-mon[65866]: overall HEALTH_OK 2024-05-06T13:57:59.514 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:59 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:57:59.514 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:57:59 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:57:59.802 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T13:58:00.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:00 smithi077 ceph-mon[84247]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:58:00.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:00 smithi077 ceph-mon[89212]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:58:00.499 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T13:58:00.499 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (2m) 14s ago 25m 19.5M - 0.25.0 c8568f914cd2 997175e3e64e 2024-05-06T13:58:00.499 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (2m) 14s ago 24m 80.6M - 9.4.12 2bacad6d85d8 140bb9a2760a 2024-05-06T13:58:00.499 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (3m) 14s ago 24m 56.1M - 3.5 e1d6a67b021e 9353a303fb55 2024-05-06T13:58:00.499 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283,8765 running (4m) 14s ago 29m 447M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 125521be53a3 2024-05-06T13:58:00.499 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (13m) 14s ago 31m 474M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T13:58:00.499 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (28s) 14s ago 31m 28.4M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 5e50731ccbe8 2024-05-06T13:58:00.499 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (78s) 14s ago 29m 36.3M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 61cd48b3d0ff 2024-05-06T13:58:00.499 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 starting - - - 2048M 2024-05-06T13:58:00.499 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (3m) 14s ago 25m 15.9M - 1.5.0 0da6a335fe13 67f45fa4968b 2024-05-06T13:58:00.499 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (3m) 14s ago 25m 16.4M - 1.5.0 0da6a335fe13 02081b71a5dd 2024-05-06T13:58:00.500 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (29m) 14s ago 29m 54.4M 2992M 17.2.0 e1d6a67b021e 2ccee2bb6e96 2024-05-06T13:58:00.500 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (28m) 14s ago 28m 58.7M 2992M 17.2.0 e1d6a67b021e 1d9e6609b585 2024-05-06T13:58:00.500 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (28m) 14s ago 28m 52.6M 2992M 17.2.0 e1d6a67b021e 37a61d295111 2024-05-06T13:58:00.500 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (28m) 14s ago 28m 52.8M 2992M 17.2.0 e1d6a67b021e 8feb7c077aaa 2024-05-06T13:58:00.500 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (27m) 14s ago 27m 58.5M 3248M 17.2.0 e1d6a67b021e f865acc44a06 2024-05-06T13:58:00.500 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (27m) 14s ago 27m 52.8M 3248M 17.2.0 e1d6a67b021e a55bd4122f6f 2024-05-06T13:58:00.500 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (27m) 14s ago 27m 51.4M 3248M 17.2.0 e1d6a67b021e 362938a0b357 2024-05-06T13:58:00.500 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (26m) 14s ago 26m 54.1M 3248M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T13:58:00.500 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (3m) 14s ago 26m 60.1M - 2.43.0 a07b618ecd1d 10d5c513bb9b 2024-05-06T13:58:00.500 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (24m) 14s ago 24m 78.0M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T13:58:00.500 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (24m) 14s ago 24m 77.2M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T13:58:00.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:00 smithi170 ceph-mon[65866]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-06T13:58:00.904 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:58:00.904 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-05-06T13:58:00.904 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T13:58:00.904 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:58:00.904 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-05-06T13:58:00.904 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T13:58:00.904 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:58:00.905 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-05-06T13:58:00.905 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-06T13:58:00.905 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:58:00.905 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw": { 2024-05-06T13:58:00.905 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T13:58:00.905 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:58:00.905 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-05-06T13:58:00.905 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-05-06T13:58:00.905 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 5 2024-05-06T13:58:00.905 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T13:58:00.905 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:58:01.250 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:58:01.250 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T13:58:01.251 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-05-06T13:58:01.251 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi077", 2024-05-06T13:58:01.251 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [], 2024-05-06T13:58:01.251 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "1/2 daemons upgraded", 2024-05-06T13:58:01.251 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "Currently upgrading mon daemons", 2024-05-06T13:58:01.251 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-05-06T13:58:01.251 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:58:01.513 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:01 smithi077 ceph-mon[89212]: from='client.54138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:58:01.513 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:01 smithi077 ceph-mon[89212]: from='client.54144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:58:01.513 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:01 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/3129127689' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:01.513 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:01 smithi077 ceph-mon[84247]: from='client.54138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:58:01.513 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:01 smithi077 ceph-mon[84247]: from='client.54144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:58:01.513 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:01 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/3129127689' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:01.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:01 smithi170 ceph-mon[65866]: from='client.54138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:58:01.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:01 smithi170 ceph-mon[65866]: from='client.54144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:58:01.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:01 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/3129127689' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:02.102 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:58:01 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:58:01.804+0000 7f5cd1795700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-05-06T13:58:02.408 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:02 smithi077 ceph-mon[84247]: from='client.54150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:58:02.409 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:02 smithi077 ceph-mon[84247]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:02.409 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:02 smithi077 ceph-mon[89212]: from='client.54150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:58:02.409 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:02 smithi077 ceph-mon[89212]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:02.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:02 smithi170 ceph-mon[65866]: from='client.54150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:58:02.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:02 smithi170 ceph-mon[65866]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:03.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:03 smithi077 ceph-mon[84247]: from='client.54162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:58:03.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:03 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:58:03.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:03 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:03.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:03 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:03.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:03 smithi077 ceph-mon[89212]: from='client.54162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:58:03.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:03 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:58:03.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:03 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:03.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:03 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:03.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:03 smithi170 ceph-mon[65866]: from='client.54162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:58:03.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:03 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:58:03.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:03 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:03.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:03 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:04.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:58:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:58:03] "GET /metrics HTTP/1.1" 200 38012 "" "Prometheus/2.43.0" 2024-05-06T13:58:04.390 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:04 smithi077 ceph-mon[89212]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:04.390 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:04 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:04.390 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:04 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:04.390 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:04 smithi077 ceph-mon[84247]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:04.390 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:04 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:04.390 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:04 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:04.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:04 smithi170 ceph-mon[65866]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:04.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:04 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:04.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:04 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:06.456 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:06 smithi077 ceph-mon[84247]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:06.457 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:06 smithi077 ceph-mon[89212]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:06.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:06 smithi170 ceph-mon[65866]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:08.186 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:08 smithi077 ceph-mon[84247]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:08.187 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:08 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:08.187 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:08 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:08.187 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:08 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:08.187 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:08 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:58:08.187 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:08 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:08.187 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:08 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:58:08.187 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:08 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T13:58:08.187 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:08 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:08.187 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:08 smithi077 ceph-mon[89212]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:08.187 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:08 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:08.188 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:08 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:08.188 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:08 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:08.188 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:08 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:58:08.188 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:08 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:08.188 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:08 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:58:08.188 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:08 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T13:58:08.188 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:08 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:08.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:08 smithi170 ceph-mon[65866]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:08.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:08 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:08.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:08 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:08.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:08 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:08.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:08 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:58:08.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:08 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:08.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:08 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:58:08.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:08 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T13:58:08.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:08 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:09.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:09 smithi077 ceph-mon[84247]: Reconfiguring mon.a (monmap changed)... 2024-05-06T13:58:09.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:09 smithi077 ceph-mon[84247]: Reconfiguring daemon mon.a on smithi077 2024-05-06T13:58:09.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:09 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:09.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:09 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:09.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:09 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T13:58:09.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:09 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T13:58:09.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:09 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:09.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:09 smithi077 ceph-mon[89212]: Reconfiguring mon.a (monmap changed)... 2024-05-06T13:58:09.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:09 smithi077 ceph-mon[89212]: Reconfiguring daemon mon.a on smithi077 2024-05-06T13:58:09.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:09 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:09.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:09 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:09.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:09 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T13:58:09.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:09 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T13:58:09.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:09 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:09.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:09 smithi170 ceph-mon[65866]: Reconfiguring mon.a (monmap changed)... 2024-05-06T13:58:09.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:09 smithi170 ceph-mon[65866]: Reconfiguring daemon mon.a on smithi077 2024-05-06T13:58:09.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:09 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:09.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:09 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:09.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:09 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T13:58:09.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:09 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T13:58:09.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:09 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:10.463 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:10 smithi077 ceph-mon[84247]: Reconfiguring mgr.y (monmap changed)... 2024-05-06T13:58:10.464 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:10 smithi077 ceph-mon[84247]: Reconfiguring daemon mgr.y on smithi077 2024-05-06T13:58:10.464 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:10 smithi077 ceph-mon[84247]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:10.464 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:10 smithi077 ceph-mon[89212]: Reconfiguring mgr.y (monmap changed)... 2024-05-06T13:58:10.464 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:10 smithi077 ceph-mon[89212]: Reconfiguring daemon mgr.y on smithi077 2024-05-06T13:58:10.464 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:10 smithi077 ceph-mon[89212]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:10.478 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:10 smithi170 ceph-mon[65866]: Reconfiguring mgr.y (monmap changed)... 2024-05-06T13:58:10.478 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:10 smithi170 ceph-mon[65866]: Reconfiguring daemon mgr.y on smithi077 2024-05-06T13:58:10.478 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:10 smithi170 ceph-mon[65866]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:11.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:11 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:11.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:11 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:11.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:11 smithi077 ceph-mon[89212]: Reconfiguring mon.c (monmap changed)... 2024-05-06T13:58:11.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:11 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:58:11.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:11 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T13:58:11.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:11 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:11.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:11 smithi077 ceph-mon[89212]: Reconfiguring daemon mon.c on smithi077 2024-05-06T13:58:11.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:11 smithi077 ceph-mon[89212]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:11.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:11 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:58:11.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:11 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:11.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:11 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:11.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:11 smithi077 ceph-mon[84247]: Reconfiguring mon.c (monmap changed)... 2024-05-06T13:58:11.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:11 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:58:11.804 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:11 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T13:58:11.804 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:11 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:11.804 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:11 smithi077 ceph-mon[84247]: Reconfiguring daemon mon.c on smithi077 2024-05-06T13:58:11.804 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:11 smithi077 ceph-mon[84247]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:11.804 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:11 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:58:11.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:11 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:11.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:11 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:11.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:11 smithi170 ceph-mon[65866]: Reconfiguring mon.c (monmap changed)... 2024-05-06T13:58:11.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:11 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:58:11.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:11 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T13:58:11.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:11 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:11.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:11 smithi170 ceph-mon[65866]: Reconfiguring daemon mon.c on smithi077 2024-05-06T13:58:11.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:11 smithi170 ceph-mon[65866]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:11.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:11 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:58:12.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:12 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:58:12.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:12 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:12.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:12 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:12.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:12 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-06T13:58:12.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:12 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:12.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:12 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:58:12.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:12 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:12.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:12 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:12.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:12 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-06T13:58:12.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:12 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:12.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:12 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:58:12.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:12 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:12.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:12 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:12.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:12 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-06T13:58:12.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:12 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:13.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:13 smithi077 ceph-mon[84247]: Reconfiguring osd.0 (monmap changed)... 2024-05-06T13:58:13.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:13 smithi077 ceph-mon[84247]: Reconfiguring daemon osd.0 on smithi077 2024-05-06T13:58:13.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:13 smithi077 ceph-mon[84247]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:13.803 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:58:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:58:13] "GET /metrics HTTP/1.1" 200 38023 "" "Prometheus/2.43.0" 2024-05-06T13:58:13.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:13 smithi077 ceph-mon[89212]: Reconfiguring osd.0 (monmap changed)... 2024-05-06T13:58:13.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:13 smithi077 ceph-mon[89212]: Reconfiguring daemon osd.0 on smithi077 2024-05-06T13:58:13.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:13 smithi077 ceph-mon[89212]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:13.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:13 smithi170 ceph-mon[65866]: Reconfiguring osd.0 (monmap changed)... 2024-05-06T13:58:13.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:13 smithi170 ceph-mon[65866]: Reconfiguring daemon osd.0 on smithi077 2024-05-06T13:58:13.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:13 smithi170 ceph-mon[65866]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:15.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:14 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:15.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:14 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:15.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:14 smithi077 ceph-mon[84247]: Reconfiguring osd.1 (monmap changed)... 2024-05-06T13:58:15.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:14 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-06T13:58:15.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:14 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:15.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:14 smithi077 ceph-mon[84247]: Reconfiguring daemon osd.1 on smithi077 2024-05-06T13:58:15.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:14 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:15.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:14 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:15.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:14 smithi077 ceph-mon[89212]: Reconfiguring osd.1 (monmap changed)... 2024-05-06T13:58:15.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:14 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-06T13:58:15.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:14 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:15.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:14 smithi077 ceph-mon[89212]: Reconfiguring daemon osd.1 on smithi077 2024-05-06T13:58:15.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:14 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:15.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:14 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:15.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:14 smithi170 ceph-mon[65866]: Reconfiguring osd.1 (monmap changed)... 2024-05-06T13:58:15.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:14 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-06T13:58:15.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:14 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:15.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:14 smithi170 ceph-mon[65866]: Reconfiguring daemon osd.1 on smithi077 2024-05-06T13:58:16.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:15 smithi077 ceph-mon[89212]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:16.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:15 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:16.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:15 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:16.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:15 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-06T13:58:16.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:15 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:16.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:15 smithi077 ceph-mon[84247]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:16.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:15 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:16.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:15 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:16.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:15 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-06T13:58:16.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:15 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:16.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:15 smithi170 ceph-mon[65866]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:16.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:15 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:16.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:15 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:16.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:15 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-06T13:58:16.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:15 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:17.000 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:16 smithi077 ceph-mon[84247]: Reconfiguring osd.2 (monmap changed)... 2024-05-06T13:58:17.000 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:16 smithi077 ceph-mon[84247]: Reconfiguring daemon osd.2 on smithi077 2024-05-06T13:58:17.001 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:16 smithi077 ceph-mon[89212]: Reconfiguring osd.2 (monmap changed)... 2024-05-06T13:58:17.001 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:16 smithi077 ceph-mon[89212]: Reconfiguring daemon osd.2 on smithi077 2024-05-06T13:58:17.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:16 smithi170 ceph-mon[65866]: Reconfiguring osd.2 (monmap changed)... 2024-05-06T13:58:17.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:16 smithi170 ceph-mon[65866]: Reconfiguring daemon osd.2 on smithi077 2024-05-06T13:58:18.175 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:17 smithi077 ceph-mon[84247]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:18.175 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:17 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:18.175 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:17 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:18.175 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:17 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-06T13:58:18.175 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:17 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:18.175 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:17 smithi077 ceph-mon[89212]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:18.176 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:17 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:18.176 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:17 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:18.176 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:17 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-06T13:58:18.176 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:17 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:18.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:17 smithi170 ceph-mon[65866]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:18.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:17 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:18.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:17 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:18.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:17 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-06T13:58:18.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:17 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:19.142 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:18 smithi077 ceph-mon[84247]: Reconfiguring osd.3 (monmap changed)... 2024-05-06T13:58:19.142 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:18 smithi077 ceph-mon[84247]: Reconfiguring daemon osd.3 on smithi077 2024-05-06T13:58:19.142 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:18 smithi077 ceph-mon[89212]: Reconfiguring osd.3 (monmap changed)... 2024-05-06T13:58:19.143 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:18 smithi077 ceph-mon[89212]: Reconfiguring daemon osd.3 on smithi077 2024-05-06T13:58:19.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:18 smithi170 ceph-mon[65866]: Reconfiguring osd.3 (monmap changed)... 2024-05-06T13:58:19.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:18 smithi170 ceph-mon[65866]: Reconfiguring daemon osd.3 on smithi077 2024-05-06T13:58:20.187 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:19 smithi077 ceph-mon[84247]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:20.188 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:19 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:20.188 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:19 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:20.188 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:19 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi077.xctpgy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T13:58:20.188 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:19 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:20.188 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:19 smithi077 ceph-mon[89212]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:20.188 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:19 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:20.188 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:19 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:20.188 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:19 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi077.xctpgy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T13:58:20.188 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:19 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:20.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:19 smithi170 ceph-mon[65866]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:20.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:19 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:20.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:19 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:20.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:19 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi077.xctpgy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T13:58:20.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:19 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:21.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:20 smithi077 ceph-mon[84247]: Reconfiguring rgw.foo.smithi077.xctpgy (monmap changed)... 2024-05-06T13:58:21.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:20 smithi077 ceph-mon[84247]: Reconfiguring daemon rgw.foo.smithi077.xctpgy on smithi077 2024-05-06T13:58:21.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:20 smithi077 ceph-mon[89212]: Reconfiguring rgw.foo.smithi077.xctpgy (monmap changed)... 2024-05-06T13:58:21.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:20 smithi077 ceph-mon[89212]: Reconfiguring daemon rgw.foo.smithi077.xctpgy on smithi077 2024-05-06T13:58:21.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:20 smithi170 ceph-mon[65866]: Reconfiguring rgw.foo.smithi077.xctpgy (monmap changed)... 2024-05-06T13:58:21.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:20 smithi170 ceph-mon[65866]: Reconfiguring daemon rgw.foo.smithi077.xctpgy on smithi077 2024-05-06T13:58:22.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:21 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:22.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:21 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:22.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:21 smithi077 ceph-mon[84247]: Reconfiguring mon.b (monmap changed)... 2024-05-06T13:58:22.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:21 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:58:22.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:21 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T13:58:22.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:21 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:22.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:21 smithi077 ceph-mon[84247]: Reconfiguring daemon mon.b on smithi170 2024-05-06T13:58:22.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:21 smithi077 ceph-mon[84247]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:22.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:21 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:22.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:21 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:22.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:21 smithi077 ceph-mon[89212]: Reconfiguring mon.b (monmap changed)... 2024-05-06T13:58:22.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:21 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:58:22.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:21 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T13:58:22.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:21 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:22.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:21 smithi077 ceph-mon[89212]: Reconfiguring daemon mon.b on smithi170 2024-05-06T13:58:22.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:21 smithi077 ceph-mon[89212]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:22.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:21 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:22.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:21 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:22.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:21 smithi170 ceph-mon[65866]: Reconfiguring mon.b (monmap changed)... 2024-05-06T13:58:22.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:21 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-06T13:58:22.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:21 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-06T13:58:22.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:21 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:22.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:21 smithi170 ceph-mon[65866]: Reconfiguring daemon mon.b on smithi170 2024-05-06T13:58:22.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:21 smithi170 ceph-mon[65866]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:23.218 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:22 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:58:23.218 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:22 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:23.218 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:22 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:23.218 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:22 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T13:58:23.218 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:22 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T13:58:23.218 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:22 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:23.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:22 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:58:23.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:22 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:23.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:22 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:23.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:22 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T13:58:23.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:22 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T13:58:23.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:22 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:23.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:22 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:58:23.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:22 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:23.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:22 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:23.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:22 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-06T13:58:23.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:22 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-06T13:58:23.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:22 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:24.054 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:58:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:58:23] "GET /metrics HTTP/1.1" 200 38023 "" "Prometheus/2.43.0" 2024-05-06T13:58:24.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:23 smithi077 ceph-mon[84247]: Reconfiguring mgr.x (monmap changed)... 2024-05-06T13:58:24.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:23 smithi077 ceph-mon[84247]: Reconfiguring daemon mgr.x on smithi170 2024-05-06T13:58:24.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:23 smithi077 ceph-mon[84247]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:24.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:23 smithi077 ceph-mon[89212]: Reconfiguring mgr.x (monmap changed)... 2024-05-06T13:58:24.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:23 smithi077 ceph-mon[89212]: Reconfiguring daemon mgr.x on smithi170 2024-05-06T13:58:24.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:23 smithi077 ceph-mon[89212]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:24.059 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:23 smithi170 ceph-mon[65866]: Reconfiguring mgr.x (monmap changed)... 2024-05-06T13:58:24.059 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:23 smithi170 ceph-mon[65866]: Reconfiguring daemon mgr.x on smithi170 2024-05-06T13:58:24.059 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:23 smithi170 ceph-mon[65866]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:25.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:25.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:25.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:25 smithi170 ceph-mon[65866]: Reconfiguring osd.4 (monmap changed)... 2024-05-06T13:58:25.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-06T13:58:25.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:25.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:25 smithi170 ceph-mon[65866]: Reconfiguring daemon osd.4 on smithi170 2024-05-06T13:58:25.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:25.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:25.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:25 smithi077 ceph-mon[84247]: Reconfiguring osd.4 (monmap changed)... 2024-05-06T13:58:25.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-06T13:58:25.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:25.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:25 smithi077 ceph-mon[84247]: Reconfiguring daemon osd.4 on smithi170 2024-05-06T13:58:25.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:25.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:25.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:25 smithi077 ceph-mon[89212]: Reconfiguring osd.4 (monmap changed)... 2024-05-06T13:58:25.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-06T13:58:25.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:25.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:25 smithi077 ceph-mon[89212]: Reconfiguring daemon osd.4 on smithi170 2024-05-06T13:58:26.402 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:26 smithi170 ceph-mon[65866]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:26.402 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:26 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:26.402 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:26 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:26.402 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:26 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-06T13:58:26.402 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:26 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:26.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:26 smithi077 ceph-mon[84247]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:26.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:26 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:26.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:26 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:26.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:26 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-06T13:58:26.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:26 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:26.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:26 smithi077 ceph-mon[89212]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:26.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:26 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:26.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:26 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:26.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:26 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-06T13:58:26.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:26 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:27.459 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:27 smithi170 ceph-mon[65866]: Reconfiguring osd.5 (monmap changed)... 2024-05-06T13:58:27.459 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:27 smithi170 ceph-mon[65866]: Reconfiguring daemon osd.5 on smithi170 2024-05-06T13:58:27.459 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:27 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:58:27.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:27 smithi077 ceph-mon[84247]: Reconfiguring osd.5 (monmap changed)... 2024-05-06T13:58:27.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:27 smithi077 ceph-mon[84247]: Reconfiguring daemon osd.5 on smithi170 2024-05-06T13:58:27.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:27 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:58:27.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:27 smithi077 ceph-mon[89212]: Reconfiguring osd.5 (monmap changed)... 2024-05-06T13:58:27.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:27 smithi077 ceph-mon[89212]: Reconfiguring daemon osd.5 on smithi170 2024-05-06T13:58:27.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:27 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:58:28.454 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:28 smithi170 ceph-mon[65866]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:28.454 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:28.454 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:28.454 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-06T13:58:28.454 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:28.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:28 smithi077 ceph-mon[84247]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:28.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:28.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:28.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-06T13:58:28.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:28.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:28 smithi077 ceph-mon[89212]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:28.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:28.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:28.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-06T13:58:28.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:29.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:29 smithi170 ceph-mon[65866]: Reconfiguring osd.6 (monmap changed)... 2024-05-06T13:58:29.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:29 smithi170 ceph-mon[65866]: Reconfiguring daemon osd.6 on smithi170 2024-05-06T13:58:29.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:29.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:29.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:29 smithi077 ceph-mon[84247]: Reconfiguring osd.6 (monmap changed)... 2024-05-06T13:58:29.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:29 smithi077 ceph-mon[84247]: Reconfiguring daemon osd.6 on smithi170 2024-05-06T13:58:29.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:29.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:29.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:29 smithi077 ceph-mon[89212]: Reconfiguring osd.6 (monmap changed)... 2024-05-06T13:58:29.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:29 smithi077 ceph-mon[89212]: Reconfiguring daemon osd.6 on smithi170 2024-05-06T13:58:29.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:29.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:30.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:30 smithi170 ceph-mon[65866]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:30.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:30 smithi170 ceph-mon[65866]: Reconfiguring osd.7 (monmap changed)... 2024-05-06T13:58:30.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:30 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-06T13:58:30.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:30 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:30.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:30 smithi170 ceph-mon[65866]: Reconfiguring daemon osd.7 on smithi170 2024-05-06T13:58:30.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:30 smithi077 ceph-mon[84247]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:30.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:30 smithi077 ceph-mon[84247]: Reconfiguring osd.7 (monmap changed)... 2024-05-06T13:58:30.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:30 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-06T13:58:30.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:30 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:30.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:30 smithi077 ceph-mon[84247]: Reconfiguring daemon osd.7 on smithi170 2024-05-06T13:58:30.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:30 smithi077 ceph-mon[89212]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:30.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:30 smithi077 ceph-mon[89212]: Reconfiguring osd.7 (monmap changed)... 2024-05-06T13:58:30.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:30 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-06T13:58:30.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:30 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:30.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:30 smithi077 ceph-mon[89212]: Reconfiguring daemon osd.7 on smithi170 2024-05-06T13:58:31.610 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T13:58:32.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:31 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:32.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:31 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:32.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:31 smithi077 ceph-mon[84247]: Reconfiguring rgw.foo.smithi170.rackmk (monmap changed)... 2024-05-06T13:58:32.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:31 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi170.rackmk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T13:58:32.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:31 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:32.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:31 smithi077 ceph-mon[84247]: Reconfiguring daemon rgw.foo.smithi170.rackmk on smithi170 2024-05-06T13:58:32.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:31 smithi077 ceph-mon[84247]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:32.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:31 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:32.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:31 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:32.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:31 smithi077 ceph-mon[89212]: Reconfiguring rgw.foo.smithi170.rackmk (monmap changed)... 2024-05-06T13:58:32.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:31 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi170.rackmk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T13:58:32.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:31 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:32.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:31 smithi077 ceph-mon[89212]: Reconfiguring daemon rgw.foo.smithi170.rackmk on smithi170 2024-05-06T13:58:32.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:31 smithi077 ceph-mon[89212]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:32.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:31 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:32.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:31 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:32.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:31 smithi170 ceph-mon[65866]: Reconfiguring rgw.foo.smithi170.rackmk (monmap changed)... 2024-05-06T13:58:32.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:31 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi170.rackmk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T13:58:32.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:31 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:32.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:31 smithi170 ceph-mon[65866]: Reconfiguring daemon rgw.foo.smithi170.rackmk on smithi170 2024-05-06T13:58:32.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:31 smithi170 ceph-mon[65866]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:32.273 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T13:58:32.273 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (3m) 29s ago 25m 19.5M - 0.25.0 c8568f914cd2 997175e3e64e 2024-05-06T13:58:32.273 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (2m) 46s ago 25m 80.6M - 9.4.12 2bacad6d85d8 140bb9a2760a 2024-05-06T13:58:32.273 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (3m) 29s ago 24m 56.5M - 3.5 e1d6a67b021e 9353a303fb55 2024-05-06T13:58:32.273 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283,8765 running (4m) 46s ago 30m 447M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 125521be53a3 2024-05-06T13:58:32.273 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (14m) 29s ago 31m 493M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T13:58:32.273 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (59s) 29s ago 31m 37.8M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 5e50731ccbe8 2024-05-06T13:58:32.273 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (110s) 46s ago 30m 36.3M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 61cd48b3d0ff 2024-05-06T13:58:32.273 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (35s) 29s ago 30m 20.0M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 8a473cc2ffd1 2024-05-06T13:58:32.273 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (4m) 29s ago 26m 16.6M - 1.5.0 0da6a335fe13 67f45fa4968b 2024-05-06T13:58:32.273 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (4m) 46s ago 26m 16.4M - 1.5.0 0da6a335fe13 02081b71a5dd 2024-05-06T13:58:32.273 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (29m) 29s ago 29m 54.4M 2992M 17.2.0 e1d6a67b021e 2ccee2bb6e96 2024-05-06T13:58:32.274 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (29m) 29s ago 29m 58.7M 2992M 17.2.0 e1d6a67b021e 1d9e6609b585 2024-05-06T13:58:32.274 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (29m) 29s ago 29m 52.8M 2992M 17.2.0 e1d6a67b021e 37a61d295111 2024-05-06T13:58:32.274 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (28m) 29s ago 28m 53.0M 2992M 17.2.0 e1d6a67b021e 8feb7c077aaa 2024-05-06T13:58:32.274 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (28m) 46s ago 28m 58.5M 3248M 17.2.0 e1d6a67b021e f865acc44a06 2024-05-06T13:58:32.274 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (27m) 46s ago 27m 52.8M 3248M 17.2.0 e1d6a67b021e a55bd4122f6f 2024-05-06T13:58:32.274 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (27m) 46s ago 27m 51.4M 3248M 17.2.0 e1d6a67b021e 362938a0b357 2024-05-06T13:58:32.274 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (27m) 46s ago 27m 54.1M 3248M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T13:58:32.274 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (3m) 46s ago 26m 60.1M - 2.43.0 a07b618ecd1d 10d5c513bb9b 2024-05-06T13:58:32.274 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (24m) 29s ago 24m 78.4M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T13:58:32.274 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (24m) 46s ago 24m 77.2M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T13:58:32.760 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='client.54168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:58:32.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='client.64108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:58:32.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:58:32.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:32.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:32.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:58:32.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:32.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-06T13:58:32.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-05-06T13:58:32.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-06T13:58:32.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-05-06T13:58:32.761 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.762 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:32.762 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.762 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:32.762 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.762 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:32.762 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.762 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-06T13:58:32.762 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-06T13:58:32.762 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.762 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:32.762 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.762 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:32.762 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.762 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:32.763 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.763 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='client.54168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:58:32.763 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='client.64108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:58:32.763 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:58:32.763 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:32.763 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:32.763 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:58:32.763 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.763 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.763 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:32.763 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-06T13:58:32.764 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-05-06T13:58:32.764 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-06T13:58:32.764 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-05-06T13:58:32.764 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.764 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:32.764 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.764 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:32.764 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.764 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:32.764 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.764 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-06T13:58:32.764 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-06T13:58:32.764 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.764 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:32.765 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.765 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:32.765 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.765 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:32.765 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.765 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:58:32.765 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-05-06T13:58:32.765 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T13:58:32.765 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:58:32.765 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-05-06T13:58:32.765 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T13:58:32.766 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:58:32.766 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-05-06T13:58:32.766 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-06T13:58:32.766 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:58:32.766 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw": { 2024-05-06T13:58:32.766 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T13:58:32.766 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:58:32.766 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-05-06T13:58:32.766 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-05-06T13:58:32.766 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 5 2024-05-06T13:58:32.766 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T13:58:32.767 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:58:32.768 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-06T13:58:32.768 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-06T13:58:32.768 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.768 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-06T13:58:32.768 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-06T13:58:32.768 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.768 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-06T13:58:32.768 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-06T13:58:32.768 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.769 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-06T13:58:32.769 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-06T13:58:32.769 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.769 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-06T13:58:32.769 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-06T13:58:32.769 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.769 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-06T13:58:32.769 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-06T13:58:32.769 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-05-06T13:58:32.769 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-05-06T13:58:32.769 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:58:32.770 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-05-06T13:58:32.770 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-05-06T13:58:32.770 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-05-06T13:58:32.770 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-06T13:58:32.771 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-06T13:58:32.771 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.771 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-06T13:58:32.771 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-06T13:58:32.771 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.771 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-06T13:58:32.771 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-06T13:58:32.771 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.771 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-06T13:58:32.771 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-06T13:58:32.771 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.771 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-06T13:58:32.772 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-06T13:58:32.772 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:32.772 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-06T13:58:32.772 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-06T13:58:32.772 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-05-06T13:58:32.772 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-05-06T13:58:32.772 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:58:32.772 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-05-06T13:58:32.772 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-05-06T13:58:32.772 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-05-06T13:58:32.772 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-05-06T13:58:32.772 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-05-06T13:58:32.772 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-05-06T13:58:32.773 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-05-06T13:58:32.773 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-05-06T13:58:32.773 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-05-06T13:58:32.773 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:58:32.773 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-05-06T13:58:32.773 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-05-06T13:58:32.773 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-05-06T13:58:32.773 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-05-06T13:58:32.773 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-05-06T13:58:32.773 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-05-06T13:58:32.773 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-05-06T13:58:32.774 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:58:32.774 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:58:32.774 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:58:32.774 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:58:32.774 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:58:32.774 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:58:32.774 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-05-06T13:58:32.774 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-05-06T13:58:32.774 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-05-06T13:58:32.775 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-05-06T13:58:32.775 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-05-06T13:58:32.775 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-05-06T13:58:32.775 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-05-06T13:58:32.775 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-05-06T13:58:32.775 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:58:32.775 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-05-06T13:58:32.775 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-05-06T13:58:32.775 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-05-06T13:58:32.775 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-05-06T13:58:32.776 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-05-06T13:58:32.776 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-05-06T13:58:32.776 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-05-06T13:58:32.776 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:58:32.776 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:58:32.776 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:58:32.776 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:58:32.776 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:58:32.776 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:58:32.776 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-05-06T13:58:32.776 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-05-06T13:58:33.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='client.54168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:58:33.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='client.64108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:58:33.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:58:33.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:33.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:33.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:58:33.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:33.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:33.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:33.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-06T13:58:33.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-05-06T13:58:33.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-06T13:58:33.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-05-06T13:58:33.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:33.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:33.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:33.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:33.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:33.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:33.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:33.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-06T13:58:33.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-06T13:58:33.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:33.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:33.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:33.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:33.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:33.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:33.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:33.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-06T13:58:33.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-06T13:58:33.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:33.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-06T13:58:33.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-06T13:58:33.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:33.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-06T13:58:33.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-06T13:58:33.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:33.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-06T13:58:33.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-06T13:58:33.080 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:33.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-06T13:58:33.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-06T13:58:33.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:33.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-05-06T13:58:33.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-05-06T13:58:33.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-05-06T13:58:33.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-05-06T13:58:33.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:58:33.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-05-06T13:58:33.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-05-06T13:58:33.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-05-06T13:58:33.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-05-06T13:58:33.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-05-06T13:58:33.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-05-06T13:58:33.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-05-06T13:58:33.081 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-05-06T13:58:33.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-05-06T13:58:33.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:58:33.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-05-06T13:58:33.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-05-06T13:58:33.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-05-06T13:58:33.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-05-06T13:58:33.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-05-06T13:58:33.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-05-06T13:58:33.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-05-06T13:58:33.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:58:33.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:58:33.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:58:33.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:58:33.082 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:58:33.083 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T13:58:33.083 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-05-06T13:58:33.083 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-05-06T13:58:33.148 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:58:33.148 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": null, 2024-05-06T13:58:33.148 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": false, 2024-05-06T13:58:33.148 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "", 2024-05-06T13:58:33.148 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [], 2024-05-06T13:58:33.148 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": null, 2024-05-06T13:58:33.148 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "", 2024-05-06T13:58:33.148 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-05-06T13:58:33.149 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:58:34.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:58:33 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:58:33] "GET /metrics HTTP/1.1" 200 38020 "" "Prometheus/2.43.0" 2024-05-06T13:58:34.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: from='client.54180 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:58:34.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all mon 2024-05-06T13:58:34.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all crash 2024-05-06T13:58:34.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all mds 2024-05-06T13:58:34.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all rbd-mirror 2024-05-06T13:58:34.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all cephfs-mirror 2024-05-06T13:58:34.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all ceph-exporter 2024-05-06T13:58:34.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all nfs 2024-05-06T13:58:34.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all nvmeof 2024-05-06T13:58:34.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all node-exporter 2024-05-06T13:58:34.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all prometheus 2024-05-06T13:58:34.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all alertmanager 2024-05-06T13:58:34.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all grafana 2024-05-06T13:58:34.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all loki 2024-05-06T13:58:34.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all promtail 2024-05-06T13:58:34.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: Upgrade: Finalizing container_image settings 2024-05-06T13:58:34.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: Upgrade: Complete! 2024-05-06T13:58:34.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/4029689178' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:34.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:34.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: from='client.54192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:58:34.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:34.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:34.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:34.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:58:34.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:34.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:34.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:58:34.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:34.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: from='client.54180 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:58:34.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all mon 2024-05-06T13:58:34.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all crash 2024-05-06T13:58:34.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all mds 2024-05-06T13:58:34.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all rbd-mirror 2024-05-06T13:58:34.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all cephfs-mirror 2024-05-06T13:58:34.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all ceph-exporter 2024-05-06T13:58:34.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all nfs 2024-05-06T13:58:34.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all nvmeof 2024-05-06T13:58:34.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all node-exporter 2024-05-06T13:58:34.055 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all prometheus 2024-05-06T13:58:34.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all alertmanager 2024-05-06T13:58:34.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all grafana 2024-05-06T13:58:34.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all loki 2024-05-06T13:58:34.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all promtail 2024-05-06T13:58:34.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: Upgrade: Finalizing container_image settings 2024-05-06T13:58:34.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: Upgrade: Complete! 2024-05-06T13:58:34.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/4029689178' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:34.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:34.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: from='client.54192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:58:34.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:34.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:34.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:34.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:58:34.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:34.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:34.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:58:34.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:33 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:34.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: from='client.54180 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:58:34.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all mon 2024-05-06T13:58:34.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all crash 2024-05-06T13:58:34.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all mds 2024-05-06T13:58:34.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all rbd-mirror 2024-05-06T13:58:34.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all cephfs-mirror 2024-05-06T13:58:34.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all ceph-exporter 2024-05-06T13:58:34.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all nfs 2024-05-06T13:58:34.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all nvmeof 2024-05-06T13:58:34.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all node-exporter 2024-05-06T13:58:34.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all prometheus 2024-05-06T13:58:34.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all alertmanager 2024-05-06T13:58:34.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all grafana 2024-05-06T13:58:34.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all loki 2024-05-06T13:58:34.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all promtail 2024-05-06T13:58:34.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: Upgrade: Finalizing container_image settings 2024-05-06T13:58:34.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: Upgrade: Complete! 2024-05-06T13:58:34.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/4029689178' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:58:34.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:34.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: from='client.54192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:58:34.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:34.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:34.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:34.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:58:34.078 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:34.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:58:34.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:58:34.079 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:33 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:36.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:36 smithi077 ceph-mon[84247]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:36.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:36 smithi077 ceph-mon[89212]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:36.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:36 smithi170 ceph-mon[65866]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:37.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:37 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:37.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:37 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:37.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:37 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:58:38.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:38 smithi077 ceph-mon[84247]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:38.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:38 smithi077 ceph-mon[89212]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:38.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:38 smithi170 ceph-mon[65866]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:40.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:40 smithi077 ceph-mon[84247]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:40.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:40 smithi077 ceph-mon[89212]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:40.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:40 smithi170 ceph-mon[65866]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:41.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:58:41.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:58:41.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:58:42.511 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:42 smithi170 ceph-mon[65866]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:42.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:42 smithi077 ceph-mon[84247]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:42.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:42 smithi077 ceph-mon[89212]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:43.411 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:43 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:58:43.411 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:43 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:58:43.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:43 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:58:44.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:58:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:58:43] "GET /metrics HTTP/1.1" 200 38008 "" "Prometheus/2.43.0" 2024-05-06T13:58:44.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:44 smithi077 ceph-mon[84247]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:44.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:44 smithi077 ceph-mon[89212]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:44.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:44 smithi170 ceph-mon[65866]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:46.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:46 smithi077 ceph-mon[84247]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:46.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:46 smithi077 ceph-mon[89212]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:46.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:46 smithi170 ceph-mon[65866]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:48.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:48 smithi077 ceph-mon[84247]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:48.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:48 smithi077 ceph-mon[89212]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:48.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:48 smithi170 ceph-mon[65866]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:50.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:50 smithi077 ceph-mon[84247]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:50.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:50 smithi077 ceph-mon[89212]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:50.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:50 smithi170 ceph-mon[65866]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:52.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:52 smithi170 ceph-mon[65866]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:52.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:52 smithi077 ceph-mon[84247]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:52.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:52 smithi077 ceph-mon[89212]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:53.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:53 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:58:53.555 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:53 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:58:53.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:53 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:58:54.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:58:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:58:53] "GET /metrics HTTP/1.1" 200 38008 "" "Prometheus/2.43.0" 2024-05-06T13:58:54.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:54 smithi077 ceph-mon[84247]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:54.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:54 smithi077 ceph-mon[89212]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:54.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:54 smithi170 ceph-mon[65866]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:58:56.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:56 smithi077 ceph-mon[84247]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:56.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:56 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:58:56.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:56 smithi077 ceph-mon[89212]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:56.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:56 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:58:56.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:56 smithi170 ceph-mon[65866]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:56.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:56 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:58:57.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:58:57 smithi077 ceph-mon[84247]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:57.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:58:57 smithi077 ceph-mon[89212]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:58:57.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:58:57 smithi170 ceph-mon[65866]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:00.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:00 smithi077 ceph-mon[84247]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:00.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:00 smithi077 ceph-mon[89212]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:00.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:00 smithi170 ceph-mon[65866]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:02.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:02 smithi077 ceph-mon[84247]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:02.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:02 smithi077 ceph-mon[89212]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:02.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:02 smithi170 ceph-mon[65866]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:03.508 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:03 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:59:03.509 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:03 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:59:03.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:03 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:59:03.803 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:59:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:59:03] "GET /metrics HTTP/1.1" 200 38018 "" "Prometheus/2.43.0" 2024-05-06T13:59:04.188 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch ps' 2024-05-06T13:59:04.494 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:04 smithi077 ceph-mon[84247]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:04.495 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:04 smithi077 ceph-mon[89212]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:04.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:04 smithi170 ceph-mon[65866]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:05.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:05 smithi077 ceph-mon[84247]: from='client.54198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:59:05.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:05 smithi077 ceph-mon[89212]: from='client.54198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:59:05.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:05 smithi170 ceph-mon[65866]: from='client.54198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:59:06.299 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T13:59:06.299 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (3m) 63s ago 26m 19.5M - 0.25.0 c8568f914cd2 997175e3e64e 2024-05-06T13:59:06.299 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (3m) 80s ago 25m 80.6M - 9.4.12 2bacad6d85d8 140bb9a2760a 2024-05-06T13:59:06.300 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (4m) 63s ago 25m 56.5M - 3.5 e1d6a67b021e 9353a303fb55 2024-05-06T13:59:06.300 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283,8765 running (5m) 80s ago 30m 447M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 125521be53a3 2024-05-06T13:59:06.300 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (14m) 63s ago 32m 493M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T13:59:06.300 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (94s) 63s ago 32m 37.8M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 5e50731ccbe8 2024-05-06T13:59:06.300 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (2m) 80s ago 31m 36.3M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 61cd48b3d0ff 2024-05-06T13:59:06.300 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (69s) 63s ago 31m 20.0M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 8a473cc2ffd1 2024-05-06T13:59:06.300 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (4m) 63s ago 27m 16.6M - 1.5.0 0da6a335fe13 67f45fa4968b 2024-05-06T13:59:06.300 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (4m) 80s ago 26m 16.4M - 1.5.0 0da6a335fe13 02081b71a5dd 2024-05-06T13:59:06.300 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (30m) 63s ago 30m 54.4M 2992M 17.2.0 e1d6a67b021e 2ccee2bb6e96 2024-05-06T13:59:06.300 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (30m) 63s ago 30m 58.7M 2992M 17.2.0 e1d6a67b021e 1d9e6609b585 2024-05-06T13:59:06.300 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (29m) 63s ago 29m 52.8M 2992M 17.2.0 e1d6a67b021e 37a61d295111 2024-05-06T13:59:06.300 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (29m) 63s ago 29m 53.0M 2992M 17.2.0 e1d6a67b021e 8feb7c077aaa 2024-05-06T13:59:06.300 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (28m) 80s ago 28m 58.5M 3248M 17.2.0 e1d6a67b021e f865acc44a06 2024-05-06T13:59:06.300 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (28m) 80s ago 28m 52.8M 3248M 17.2.0 e1d6a67b021e a55bd4122f6f 2024-05-06T13:59:06.301 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (28m) 80s ago 28m 51.4M 3248M 17.2.0 e1d6a67b021e 362938a0b357 2024-05-06T13:59:06.301 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (27m) 80s ago 27m 54.1M 3248M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T13:59:06.301 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (4m) 80s ago 27m 60.1M - 2.43.0 a07b618ecd1d 10d5c513bb9b 2024-05-06T13:59:06.301 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (25m) 63s ago 25m 78.4M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T13:59:06.301 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (25m) 80s ago 25m 77.2M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T13:59:06.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:06 smithi077 ceph-mon[84247]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:06.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:06 smithi077 ceph-mon[89212]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:06.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:06 smithi170 ceph-mon[65866]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:06.985 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph versions | jq -e '"'"'.mon | length == 1'"'"'' 2024-05-06T13:59:08.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:08 smithi077 ceph-mon[84247]: from='client.54204 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:59:08.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:08 smithi077 ceph-mon[84247]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:08.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:08 smithi077 ceph-mon[89212]: from='client.54204 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:59:08.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:08 smithi077 ceph-mon[89212]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:08.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:08 smithi170 ceph-mon[65866]: from='client.54204 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:59:08.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:08 smithi170 ceph-mon[65866]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:09.186 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T13:59:09.875 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph versions | jq -e '"'"'.mon | keys'"'"' | grep $sha1' 2024-05-06T13:59:10.462 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:10 smithi077 ceph-mon[84247]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:10.462 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:10 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/197002486' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:59:10.462 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:10 smithi077 ceph-mon[89212]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:10.462 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:10 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/197002486' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:59:10.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:10 smithi170 ceph-mon[65866]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:10.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:10 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/197002486' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:59:11.362 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:11 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:59:11.362 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:11 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:59:11.520 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:11 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:59:11.889 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)" 2024-05-06T13:59:12.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:12 smithi077 ceph-mon[84247]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:12.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:12 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/17742126' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:59:12.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:12 smithi077 ceph-mon[89212]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:12.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:12 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/17742126' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:59:12.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:12 smithi170 ceph-mon[65866]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:12.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:12 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/17742126' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:59:12.580 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 5'"'"'' 2024-05-06T13:59:13.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:13 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:59:13.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:13 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:59:13.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:13 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:59:13.997 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:59:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:59:13] "GET /metrics HTTP/1.1" 200 38016 "" "Prometheus/2.43.0" 2024-05-06T13:59:14.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:14 smithi077 ceph-mon[84247]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:14.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:14 smithi077 ceph-mon[89212]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:14.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:14 smithi170 ceph-mon[65866]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:15.468 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:15 smithi077 ceph-mon[84247]: from='client.64156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:59:15.468 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:15 smithi077 ceph-mon[84247]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:15.469 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:15 smithi077 ceph-mon[89212]: from='client.64156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:59:15.469 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:15 smithi077 ceph-mon[89212]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:15.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:15 smithi170 ceph-mon[65866]: from='client.64156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:59:15.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:15 smithi170 ceph-mon[65866]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:16.160 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T13:59:16.718 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch upgrade status' 2024-05-06T13:59:18.258 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:18 smithi077 ceph-mon[84247]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:18.259 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:18 smithi077 ceph-mon[89212]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:18.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:18 smithi170 ceph-mon[65866]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:18.684 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:59:18.685 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": null, 2024-05-06T13:59:18.685 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": false, 2024-05-06T13:59:18.685 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "", 2024-05-06T13:59:18.685 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [], 2024-05-06T13:59:18.685 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": null, 2024-05-06T13:59:18.685 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "", 2024-05-06T13:59:18.685 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-05-06T13:59:18.685 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:59:19.351 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph health detail' 2024-05-06T13:59:20.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:20 smithi077 ceph-mon[89212]: from='client.54228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:59:20.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:20 smithi077 ceph-mon[89212]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:20.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:20 smithi077 ceph-mon[84247]: from='client.54228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:59:20.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:20 smithi077 ceph-mon[84247]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:20.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:20 smithi170 ceph-mon[65866]: from='client.54228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:59:20.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:20 smithi170 ceph-mon[65866]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:21.397 INFO:teuthology.orchestra.run.smithi077.stdout:HEALTH_OK 2024-05-06T13:59:22.047 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 2' 2024-05-06T13:59:22.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:22 smithi077 ceph-mon[84247]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:22.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:22 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/1289136946' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T13:59:22.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:22 smithi077 ceph-mon[89212]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:22.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:22 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/1289136946' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T13:59:22.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:22 smithi170 ceph-mon[65866]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:22.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:22 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/1289136946' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T13:59:23.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:23 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:59:23.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:23 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:59:23.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:23 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:59:24.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:59:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:59:23] "GET /metrics HTTP/1.1" 200 38016 "" "Prometheus/2.43.0" 2024-05-06T13:59:24.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:24 smithi077 ceph-mon[84247]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:24.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:24 smithi077 ceph-mon[89212]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:24.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:24 smithi170 ceph-mon[65866]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:25.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:25 smithi077 ceph-mon[84247]: from='client.64174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:59:25.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:25 smithi077 ceph-mon[89212]: from='client.64174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:59:25.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:25 smithi170 ceph-mon[65866]: from='client.64174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:59:25.609 INFO:teuthology.orchestra.run.smithi077.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:59:26.261 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:26 smithi077 ceph-mon[89212]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:26.261 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:26 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:26.262 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:26 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:59:26.262 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:26 smithi077 ceph-mon[84247]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:26.262 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:26 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:26.262 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:26 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:59:26.284 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-05-06T13:59:26.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:26 smithi170 ceph-mon[65866]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:26.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:26 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:26.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:26 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:59:27.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:27 smithi077 ceph-mon[84247]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:59:27.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:27 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:59:27.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:27 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:59:27.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:27 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:27.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:27 smithi077 ceph-mon[89212]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:59:27.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:27 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:59:27.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:27 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:59:27.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:27 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:27.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:27 smithi170 ceph-mon[65866]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:59:27.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:27 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:59:27.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:27 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:59:27.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:27 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:28.209 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:28 smithi077 ceph-mon[84247]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:59:28.209 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:28 smithi077 ceph-mon[84247]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:28.209 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:28 smithi077 ceph-mon[89212]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:59:28.209 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:28 smithi077 ceph-mon[89212]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:28.554 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T13:59:28.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:28 smithi170 ceph-mon[65866]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:59:28.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:28 smithi170 ceph-mon[65866]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:29.214 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T13:59:29.214 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (4m) 86s ago 26m 19.5M - 0.25.0 c8568f914cd2 997175e3e64e 2024-05-06T13:59:29.214 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (3m) 103s ago 26m 80.6M - 9.4.12 2bacad6d85d8 140bb9a2760a 2024-05-06T13:59:29.214 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (4m) 86s ago 25m 56.5M - 3.5 e1d6a67b021e 9353a303fb55 2024-05-06T13:59:29.214 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283,8765 running (5m) 103s ago 31m 447M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 125521be53a3 2024-05-06T13:59:29.214 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (15m) 86s ago 32m 493M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T13:59:29.214 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (116s) 86s ago 32m 37.8M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 5e50731ccbe8 2024-05-06T13:59:29.214 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (2m) 103s ago 31m 36.3M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 61cd48b3d0ff 2024-05-06T13:59:29.214 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (92s) 86s ago 31m 20.0M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 8a473cc2ffd1 2024-05-06T13:59:29.215 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (5m) 86s ago 27m 16.6M - 1.5.0 0da6a335fe13 67f45fa4968b 2024-05-06T13:59:29.215 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (5m) 103s ago 27m 16.4M - 1.5.0 0da6a335fe13 02081b71a5dd 2024-05-06T13:59:29.215 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (30m) 86s ago 30m 54.4M 2992M 17.2.0 e1d6a67b021e 2ccee2bb6e96 2024-05-06T13:59:29.215 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (30m) 86s ago 30m 58.7M 2992M 17.2.0 e1d6a67b021e 1d9e6609b585 2024-05-06T13:59:29.215 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (30m) 86s ago 30m 52.8M 2992M 17.2.0 e1d6a67b021e 37a61d295111 2024-05-06T13:59:29.215 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (29m) 86s ago 29m 53.0M 2992M 17.2.0 e1d6a67b021e 8feb7c077aaa 2024-05-06T13:59:29.215 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (29m) 103s ago 29m 58.5M 3248M 17.2.0 e1d6a67b021e f865acc44a06 2024-05-06T13:59:29.215 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (28m) 103s ago 28m 52.8M 3248M 17.2.0 e1d6a67b021e a55bd4122f6f 2024-05-06T13:59:29.215 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (28m) 103s ago 28m 51.4M 3248M 17.2.0 e1d6a67b021e 362938a0b357 2024-05-06T13:59:29.215 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (28m) 103s ago 28m 54.1M 3248M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T13:59:29.215 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (4m) 103s ago 27m 60.1M - 2.43.0 a07b618ecd1d 10d5c513bb9b 2024-05-06T13:59:29.215 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (25m) 86s ago 25m 78.4M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T13:59:29.215 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (25m) 103s ago 25m 77.2M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T13:59:29.634 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:59:29.634 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-05-06T13:59:29.634 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T13:59:29.634 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:59:29.634 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-05-06T13:59:29.634 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T13:59:29.634 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:59:29.634 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-05-06T13:59:29.634 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-05-06T13:59:29.634 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:59:29.634 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw": { 2024-05-06T13:59:29.635 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T13:59:29.635 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T13:59:29.635 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-05-06T13:59:29.635 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-05-06T13:59:29.635 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 5 2024-05-06T13:59:29.635 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T13:59:29.635 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:59:29.986 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T13:59:29.986 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T13:59:29.986 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-05-06T13:59:29.986 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 2 daemons (2 remaining).", 2024-05-06T13:59:29.986 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [], 2024-05-06T13:59:29.986 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "0/8 daemons upgraded", 2024-05-06T13:59:29.986 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "Currently upgrading osd daemons", 2024-05-06T13:59:29.986 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-05-06T13:59:29.986 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T13:59:30.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:30 smithi077 ceph-mon[84247]: from='client.64180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:59:30.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:30 smithi077 ceph-mon[84247]: from='client.64186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:59:30.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:30 smithi077 ceph-mon[84247]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:30.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:30 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:30.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:30 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:59:30.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:30 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:59:30.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:30 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:30.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:30 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:59:30.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:30 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:30.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:30 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:59:30.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:30 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:30.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:30 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-05-06T13:59:30.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:30 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/1188683415' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:59:30.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:30 smithi077 ceph-mon[89212]: from='client.64180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:59:30.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:30 smithi077 ceph-mon[89212]: from='client.64186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:59:30.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:30 smithi077 ceph-mon[89212]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:30.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:30 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:30.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:30 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:59:30.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:30 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:59:30.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:30 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:30.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:30 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:59:30.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:30 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:30.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:30 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:59:30.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:30 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:30.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:30 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-05-06T13:59:30.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:30 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/1188683415' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:59:30.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:30 smithi170 ceph-mon[65866]: from='client.64180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:59:30.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:30 smithi170 ceph-mon[65866]: from='client.64186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:59:30.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:30 smithi170 ceph-mon[65866]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:30.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:30 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:30.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:30 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:59:30.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:30 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:59:30.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:30 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:30.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:30 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:59:30.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:30 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:30.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:30 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:59:30.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:30 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:30.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:30 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-05-06T13:59:30.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:30 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/1188683415' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:59:31.523 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:31 smithi077 ceph-mon[84247]: from='client.64192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:59:31.523 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:31 smithi077 ceph-mon[84247]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (squid) 2024-05-06T13:59:31.523 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:31 smithi077 ceph-mon[84247]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T13:59:31.523 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:31 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all mgr 2024-05-06T13:59:31.524 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:31 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all mon 2024-05-06T13:59:31.524 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:31 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all crash 2024-05-06T13:59:31.524 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:31 smithi077 ceph-mon[84247]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-05-06T13:59:31.524 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:31 smithi077 ceph-mon[84247]: Upgrade: osd.0 is safe to restart 2024-05-06T13:59:31.524 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:31 smithi077 ceph-mon[84247]: from='client.54270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:59:31.524 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:31 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:31.524 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:31 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-06T13:59:31.524 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:31 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:59:31.524 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:31 smithi077 ceph-mon[89212]: from='client.64192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:59:31.524 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:31 smithi077 ceph-mon[89212]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (squid) 2024-05-06T13:59:31.524 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:31 smithi077 ceph-mon[89212]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T13:59:31.524 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:31 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all mgr 2024-05-06T13:59:31.524 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:31 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all mon 2024-05-06T13:59:31.525 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:31 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all crash 2024-05-06T13:59:31.525 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:31 smithi077 ceph-mon[89212]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-05-06T13:59:31.525 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:31 smithi077 ceph-mon[89212]: Upgrade: osd.0 is safe to restart 2024-05-06T13:59:31.525 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:31 smithi077 ceph-mon[89212]: from='client.54270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:59:31.525 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:31 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:31.525 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:31 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-06T13:59:31.525 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:31 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:59:31.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:31 smithi170 ceph-mon[65866]: from='client.64192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:59:31.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:31 smithi170 ceph-mon[65866]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (squid) 2024-05-06T13:59:31.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:31 smithi170 ceph-mon[65866]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T13:59:31.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:31 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all mgr 2024-05-06T13:59:31.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:31 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all mon 2024-05-06T13:59:31.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:31 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all crash 2024-05-06T13:59:31.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:31 smithi170 ceph-mon[65866]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-05-06T13:59:31.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:31 smithi170 ceph-mon[65866]: Upgrade: osd.0 is safe to restart 2024-05-06T13:59:31.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:31 smithi170 ceph-mon[65866]: from='client.54270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T13:59:31.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:31 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:31.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:31 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-06T13:59:31.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:31 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:59:32.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:32 smithi077 ceph-mon[84247]: Upgrade: Updating osd.0 2024-05-06T13:59:32.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:32 smithi077 ceph-mon[84247]: Deploying daemon osd.0 on smithi077 2024-05-06T13:59:32.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:32 smithi077 ceph-mon[84247]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:32.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:32 smithi077 ceph-mon[89212]: Upgrade: Updating osd.0 2024-05-06T13:59:32.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:32 smithi077 ceph-mon[89212]: Deploying daemon osd.0 on smithi077 2024-05-06T13:59:32.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:32 smithi077 ceph-mon[89212]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:32.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:32 smithi170 ceph-mon[65866]: Upgrade: Updating osd.0 2024-05-06T13:59:32.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:32 smithi170 ceph-mon[65866]: Deploying daemon osd.0 on smithi077 2024-05-06T13:59:32.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:32 smithi170 ceph-mon[65866]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:33.931 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:59:33 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:59:33] "GET /metrics HTTP/1.1" 200 38017 "" "Prometheus/2.43.0" 2024-05-06T13:59:34.286 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:34 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:59:34.286 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:34 smithi077 ceph-mon[84247]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:34.286 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:34 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:59:34.286 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:34 smithi077 ceph-mon[89212]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:34.552 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:34 smithi077 systemd[1]: Stopping Ceph osd.0 for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:59:34.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:34 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:59:34.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:34 smithi170 ceph-mon[65866]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:35.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0[30015]: 2024-05-06T13:59:34.602+0000 7f3f6bbb3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T13:59:35.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0[30015]: 2024-05-06T13:59:34.602+0000 7f3f6bbb3700 -1 osd.0 113 *** Got signal Terminated *** 2024-05-06T13:59:35.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0[30015]: 2024-05-06T13:59:34.602+0000 7f3f6bbb3700 -1 osd.0 113 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T13:59:35.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:35 smithi077 ceph-mon[84247]: osd.0 marked itself down and dead 2024-05-06T13:59:35.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:35 smithi077 ceph-mon[89212]: osd.0 marked itself down and dead 2024-05-06T13:59:35.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:35 smithi170 ceph-mon[65866]: osd.0 marked itself down and dead 2024-05-06T13:59:35.920 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:35 smithi077 podman[96230]: 2024-05-06 13:59:35.626935437 +0000 UTC m=+1.195448010 container died 2ccee2bb6e9677c0f3ae1fee25c56557469a24b016ef87f54b441921972c4a88 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, distribution-scope=public, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, version=8, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, RELEASE=HEAD, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, vendor=Red Hat, Inc., name=centos-stream, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-05-06T13:59:36.194 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:36 smithi077 podman[96230]: 2024-05-06 13:59:36.002011393 +0000 UTC m=+1.570523957 container cleanup 2ccee2bb6e9677c0f3ae1fee25c56557469a24b016ef87f54b441921972c4a88 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, vcs-type=git, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD) 2024-05-06T13:59:36.194 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:36 smithi077 bash[96230]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0 2024-05-06T13:59:36.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:36 smithi077 ceph-mon[84247]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:36.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:36 smithi077 ceph-mon[84247]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T13:59:36.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:36 smithi077 ceph-mon[84247]: osdmap e114: 8 total, 7 up, 8 in 2024-05-06T13:59:36.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:36 smithi077 ceph-mon[89212]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:36.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:36 smithi077 ceph-mon[89212]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T13:59:36.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:36 smithi077 ceph-mon[89212]: osdmap e114: 8 total, 7 up, 8 in 2024-05-06T13:59:36.553 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:36 smithi077 podman[96248]: 2024-05-06 13:59:36.287099323 +0000 UTC m=+0.662034303 container remove 2ccee2bb6e9677c0f3ae1fee25c56557469a24b016ef87f54b441921972c4a88 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, vcs-type=git, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., version=8, ceph=True, release=754, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, name=centos-stream, io.openshift.expose-services=, distribution-scope=public, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream) 2024-05-06T13:59:36.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:36 smithi170 ceph-mon[65866]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T13:59:36.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:36 smithi170 ceph-mon[65866]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T13:59:36.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:36 smithi170 ceph-mon[65866]: osdmap e114: 8 total, 7 up, 8 in 2024-05-06T13:59:36.911 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:36 smithi077 podman[96335]: 2024-05-06 13:59:36.483317715 +0000 UTC m=+0.018324650 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:59:36.911 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:36 smithi077 podman[96335]: 2024-05-06 13:59:36.597443848 +0000 UTC m=+0.132450812 container create f5a94585497b75a5b8bcaaf04688844313e1410af131387e06d6355934a4a667 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-70ce953, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-05-06T13:59:36.911 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:36 smithi077 podman[96335]: 2024-05-06 13:59:36.824162276 +0000 UTC m=+0.359169208 container init f5a94585497b75a5b8bcaaf04688844313e1410af131387e06d6355934a4a667 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-70ce953, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-06T13:59:36.911 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:36 smithi077 podman[96335]: 2024-05-06 13:59:36.827340949 +0000 UTC m=+0.362347876 container start f5a94585497b75a5b8bcaaf04688844313e1410af131387e06d6355934a4a667 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240430, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-70ce953, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T13:59:36.911 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:36 smithi077 podman[96335]: 2024-05-06 13:59:36.910767969 +0000 UTC m=+0.445774900 container attach f5a94585497b75a5b8bcaaf04688844313e1410af131387e06d6355934a4a667 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T13:59:37.201 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:37 smithi077 podman[96335]: 2024-05-06 13:59:37.117983691 +0000 UTC m=+0.652990614 container died f5a94585497b75a5b8bcaaf04688844313e1410af131387e06d6355934a4a667 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T13:59:37.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:37 smithi077 ceph-mon[84247]: osdmap e115: 8 total, 7 up, 8 in 2024-05-06T13:59:37.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:37 smithi077 ceph-mon[84247]: pgmap v63: 161 pgs: 13 active+undersized+degraded, 21 active+undersized, 116 active+clean, 11 stale+active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 43/627 objects degraded (6.858%) 2024-05-06T13:59:37.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:37 smithi077 ceph-mon[89212]: osdmap e115: 8 total, 7 up, 8 in 2024-05-06T13:59:37.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:37 smithi077 ceph-mon[89212]: pgmap v63: 161 pgs: 13 active+undersized+degraded, 21 active+undersized, 116 active+clean, 11 stale+active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 43/627 objects degraded (6.858%) 2024-05-06T13:59:37.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:37 smithi170 ceph-mon[65866]: osdmap e115: 8 total, 7 up, 8 in 2024-05-06T13:59:37.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:37 smithi170 ceph-mon[65866]: pgmap v63: 161 pgs: 13 active+undersized+degraded, 21 active+undersized, 116 active+clean, 11 stale+active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 43/627 objects degraded (6.858%) 2024-05-06T13:59:38.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:37 smithi077 podman[96361]: 2024-05-06 13:59:37.694378864 +0000 UTC m=+0.571818497 container remove f5a94585497b75a5b8bcaaf04688844313e1410af131387e06d6355934a4a667 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-05-06T13:59:38.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:37 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.0.service: Deactivated successfully. 2024-05-06T13:59:38.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:37 smithi077 systemd[1]: Stopped Ceph osd.0 for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:59:38.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:37 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.0.service: Consumed 13.975s CPU time. 2024-05-06T13:59:38.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:37 smithi077 systemd[1]: Starting Ceph osd.0 for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T13:59:38.489 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:38 smithi077 ceph-mon[84247]: Health check failed: Degraded data redundancy: 43/627 objects degraded (6.858%), 13 pgs degraded (PG_DEGRADED) 2024-05-06T13:59:38.490 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:38 smithi077 ceph-mon[89212]: Health check failed: Degraded data redundancy: 43/627 objects degraded (6.858%), 13 pgs degraded (PG_DEGRADED) 2024-05-06T13:59:38.490 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:38 smithi077 podman[96467]: 2024-05-06 13:59:38.045921447 +0000 UTC m=+0.018220442 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:59:38.490 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:38 smithi077 podman[96467]: 2024-05-06 13:59:38.145686824 +0000 UTC m=+0.117985808 container create 585858fd0f733b9ace3cc877e2e55ec4152824d71bbbac69e65d052999949c79 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0-activate, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T13:59:38.490 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:38 smithi077 podman[96467]: 2024-05-06 13:59:38.38197443 +0000 UTC m=+0.354273420 container init 585858fd0f733b9ace3cc877e2e55ec4152824d71bbbac69e65d052999949c79 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0-activate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True) 2024-05-06T13:59:38.490 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:38 smithi077 podman[96467]: 2024-05-06 13:59:38.385666266 +0000 UTC m=+0.357965251 container start 585858fd0f733b9ace3cc877e2e55ec4152824d71bbbac69e65d052999949c79 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430) 2024-05-06T13:59:38.490 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:38 smithi077 podman[96467]: 2024-05-06 13:59:38.489909529 +0000 UTC m=+0.462208519 container attach 585858fd0f733b9ace3cc877e2e55ec4152824d71bbbac69e65d052999949c79 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0-activate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=) 2024-05-06T13:59:38.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:38 smithi170 ceph-mon[65866]: Health check failed: Degraded data redundancy: 43/627 objects degraded (6.858%), 13 pgs degraded (PG_DEGRADED) 2024-05-06T13:59:39.178 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:38 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0-activate[96482]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-06T13:59:39.178 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:38 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0-activate[96482]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-05-06T13:59:39.178 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:38 smithi077 bash[96467]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-06T13:59:39.178 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:38 smithi077 bash[96467]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-05-06T13:59:39.179 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:38 smithi077 bash[96467]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-05-06T13:59:39.179 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:38 smithi077 bash[96467]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-05-06T13:59:39.179 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:38 smithi077 bash[96467]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-05-06T13:59:39.179 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:38 smithi077 bash[96467]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-06T13:59:39.179 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:38 smithi077 bash[96467]: --> ceph-volume raw activate successful for osd ID: 0 2024-05-06T13:59:39.179 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:38 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0-activate[96482]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-05-06T13:59:39.179 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:38 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0-activate[96482]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-05-06T13:59:39.179 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:38 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0-activate[96482]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-05-06T13:59:39.179 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:38 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0-activate[96482]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-06T13:59:39.179 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:38 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0-activate[96482]: --> ceph-volume raw activate successful for osd ID: 0 2024-05-06T13:59:39.179 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:38 smithi077 podman[96467]: 2024-05-06 13:59:38.853518326 +0000 UTC m=+0.825817302 container died 585858fd0f733b9ace3cc877e2e55ec4152824d71bbbac69e65d052999949c79 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0-activate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-70ce953, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-05-06T13:59:39.437 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:39 smithi077 ceph-mon[84247]: pgmap v64: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 68/627 objects degraded (10.845%) 2024-05-06T13:59:39.438 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:39 smithi077 ceph-mon[89212]: pgmap v64: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 68/627 objects degraded (10.845%) 2024-05-06T13:59:39.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:39 smithi170 ceph-mon[65866]: pgmap v64: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 68/627 objects degraded (10.845%) 2024-05-06T13:59:39.686 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:39 smithi077 podman[96613]: 2024-05-06 13:59:39.435630294 +0000 UTC m=+0.577014900 container remove 585858fd0f733b9ace3cc877e2e55ec4152824d71bbbac69e65d052999949c79 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0-activate, RELEASE=squid-70ce953, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS) 2024-05-06T13:59:39.687 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:39 smithi077 podman[96679]: 2024-05-06 13:59:39.554713085 +0000 UTC m=+0.017859712 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T13:59:39.997 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:39 smithi077 podman[96679]: 2024-05-06 13:59:39.686895894 +0000 UTC m=+0.150042557 container create fbe876a933edd8dfb84a48e5b5e09a4a937b71085a458772846a4171c9bdf5d8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-05-06T13:59:39.998 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:39 smithi077 podman[96679]: 2024-05-06 13:59:39.897670394 +0000 UTC m=+0.360817011 container init fbe876a933edd8dfb84a48e5b5e09a4a937b71085a458772846a4171c9bdf5d8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-06T13:59:39.998 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:39 smithi077 podman[96679]: 2024-05-06 13:59:39.900600849 +0000 UTC m=+0.363747476 container start fbe876a933edd8dfb84a48e5b5e09a4a937b71085a458772846a4171c9bdf5d8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2) 2024-05-06T13:59:39.998 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:39 smithi077 ceph-osd[96698]: -- 172.21.15.77:0/2688157819 <== mon.0 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56273d31f680 con 0x56273d2e9000 2024-05-06T13:59:40.302 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:39 smithi077 bash[96679]: fbe876a933edd8dfb84a48e5b5e09a4a937b71085a458772846a4171c9bdf5d8 2024-05-06T13:59:40.302 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:40 smithi077 systemd[1]: Started Ceph osd.0 for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T13:59:40.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:40 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0[96694]: 2024-05-06T13:59:40.443+0000 7f7cc7a127c0 -1 Falling back to public interface 2024-05-06T13:59:41.151 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:41.151 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:41.151 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:41.151 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:41.151 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:41.151 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:41.151 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:41.151 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:41.151 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:41.151 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:41.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:41.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:41.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:41.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:41.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:42.531 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:42 smithi077 ceph-mon[84247]: pgmap v65: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 68/627 objects degraded (10.845%) 2024-05-06T13:59:42.532 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:42 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:42.532 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:42 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:59:42.532 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:42 smithi077 ceph-mon[89212]: pgmap v65: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 68/627 objects degraded (10.845%) 2024-05-06T13:59:42.532 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:42 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:42.532 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:42 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:59:42.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:42 smithi170 ceph-mon[65866]: pgmap v65: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 68/627 objects degraded (10.845%) 2024-05-06T13:59:42.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:42 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:42.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:42 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:59:43.416 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:43 smithi077 ceph-mon[84247]: Health check update: Degraded data redundancy: 68/627 objects degraded (10.845%), 21 pgs degraded (PG_DEGRADED) 2024-05-06T13:59:43.416 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:43 smithi077 ceph-mon[89212]: Health check update: Degraded data redundancy: 68/627 objects degraded (10.845%), 21 pgs degraded (PG_DEGRADED) 2024-05-06T13:59:43.416 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0[96694]: 2024-05-06T13:59:43.150+0000 7f7cc7a127c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-05-06T13:59:43.416 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0[96694]: 2024-05-06T13:59:43.175+0000 7f7cc7a127c0 -1 osd.0 113 log_to_monitors true 2024-05-06T13:59:43.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:43 smithi170 ceph-mon[65866]: Health check update: Degraded data redundancy: 68/627 objects degraded (10.845%), 21 pgs degraded (PG_DEGRADED) 2024-05-06T13:59:44.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:59:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:59:43] "GET /metrics HTTP/1.1" 200 38147 "" "Prometheus/2.43.0" 2024-05-06T13:59:44.552 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 13:59:44 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0[96694]: 2024-05-06T13:59:44.184+0000 7f7cb8e69700 -1 osd.0 113 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T13:59:44.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:44 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:59:44.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:44 smithi077 ceph-mon[84247]: pgmap v66: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 68/627 objects degraded (10.845%) 2024-05-06T13:59:44.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:44 smithi077 ceph-mon[84247]: from='osd.0 [v2:172.21.15.77:6802/3537669134,v1:172.21.15.77:6803/3537669134]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-06T13:59:44.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:44 smithi077 ceph-mon[84247]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-06T13:59:44.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:44 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:59:44.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:44 smithi077 ceph-mon[89212]: pgmap v66: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 68/627 objects degraded (10.845%) 2024-05-06T13:59:44.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:44 smithi077 ceph-mon[89212]: from='osd.0 [v2:172.21.15.77:6802/3537669134,v1:172.21.15.77:6803/3537669134]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-06T13:59:44.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:44 smithi077 ceph-mon[89212]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-06T13:59:44.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:44 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:59:44.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:44 smithi170 ceph-mon[65866]: pgmap v66: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 68/627 objects degraded (10.845%) 2024-05-06T13:59:44.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:44 smithi170 ceph-mon[65866]: from='osd.0 [v2:172.21.15.77:6802/3537669134,v1:172.21.15.77:6803/3537669134]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-06T13:59:44.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:44 smithi170 ceph-mon[65866]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-06T13:59:45.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:45 smithi077 ceph-mon[84247]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-06T13:59:45.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:45 smithi077 ceph-mon[84247]: osdmap e116: 8 total, 7 up, 8 in 2024-05-06T13:59:45.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:45 smithi077 ceph-mon[84247]: from='osd.0 [v2:172.21.15.77:6802/3537669134,v1:172.21.15.77:6803/3537669134]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T13:59:45.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:45 smithi077 ceph-mon[84247]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T13:59:45.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:45 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:45.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:45 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:45.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:45 smithi077 ceph-mon[89212]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-06T13:59:45.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:45 smithi077 ceph-mon[89212]: osdmap e116: 8 total, 7 up, 8 in 2024-05-06T13:59:45.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:45 smithi077 ceph-mon[89212]: from='osd.0 [v2:172.21.15.77:6802/3537669134,v1:172.21.15.77:6803/3537669134]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T13:59:45.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:45 smithi077 ceph-mon[89212]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T13:59:45.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:45 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:45.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:45 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:45.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:45 smithi170 ceph-mon[65866]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-06T13:59:45.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:45 smithi170 ceph-mon[65866]: osdmap e116: 8 total, 7 up, 8 in 2024-05-06T13:59:45.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:45 smithi170 ceph-mon[65866]: from='osd.0 [v2:172.21.15.77:6802/3537669134,v1:172.21.15.77:6803/3537669134]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T13:59:45.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:45 smithi170 ceph-mon[65866]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T13:59:45.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:45 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:45.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:45 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:46.520 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:46 smithi077 ceph-mon[89212]: pgmap v68: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 68/627 objects degraded (10.845%) 2024-05-06T13:59:46.520 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:46 smithi077 ceph-mon[89212]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T13:59:46.520 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:46 smithi077 ceph-mon[89212]: osd.0 [v2:172.21.15.77:6802/3537669134,v1:172.21.15.77:6803/3537669134] boot 2024-05-06T13:59:46.521 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:46 smithi077 ceph-mon[89212]: osdmap e117: 8 total, 8 up, 8 in 2024-05-06T13:59:46.521 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:46 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:59:46.521 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:46 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:46.521 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:46 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:46.521 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:46 smithi077 ceph-mon[89212]: osdmap e118: 8 total, 8 up, 8 in 2024-05-06T13:59:46.521 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:46 smithi077 ceph-mon[84247]: pgmap v68: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 68/627 objects degraded (10.845%) 2024-05-06T13:59:46.521 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:46 smithi077 ceph-mon[84247]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T13:59:46.521 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:46 smithi077 ceph-mon[84247]: osd.0 [v2:172.21.15.77:6802/3537669134,v1:172.21.15.77:6803/3537669134] boot 2024-05-06T13:59:46.521 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:46 smithi077 ceph-mon[84247]: osdmap e117: 8 total, 8 up, 8 in 2024-05-06T13:59:46.521 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:46 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:59:46.521 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:46 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:46.521 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:46 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:46.522 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:46 smithi077 ceph-mon[84247]: osdmap e118: 8 total, 8 up, 8 in 2024-05-06T13:59:46.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:46 smithi170 ceph-mon[65866]: pgmap v68: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 68/627 objects degraded (10.845%) 2024-05-06T13:59:46.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:46 smithi170 ceph-mon[65866]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T13:59:46.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:46 smithi170 ceph-mon[65866]: osd.0 [v2:172.21.15.77:6802/3537669134,v1:172.21.15.77:6803/3537669134] boot 2024-05-06T13:59:46.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:46 smithi170 ceph-mon[65866]: osdmap e117: 8 total, 8 up, 8 in 2024-05-06T13:59:46.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:46 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-06T13:59:46.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:46 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:46.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:46 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:46.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:46 smithi170 ceph-mon[65866]: osdmap e118: 8 total, 8 up, 8 in 2024-05-06T13:59:48.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:48 smithi077 ceph-mon[84247]: pgmap v71: 161 pgs: 9 active+undersized+degraded, 12 active+undersized, 140 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 30/627 objects degraded (4.785%) 2024-05-06T13:59:49.029 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:48 smithi077 ceph-mon[84247]: Health check update: Degraded data redundancy: 30/627 objects degraded (4.785%), 9 pgs degraded (PG_DEGRADED) 2024-05-06T13:59:49.029 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:48 smithi077 ceph-mon[89212]: pgmap v71: 161 pgs: 9 active+undersized+degraded, 12 active+undersized, 140 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 30/627 objects degraded (4.785%) 2024-05-06T13:59:49.029 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:48 smithi077 ceph-mon[89212]: Health check update: Degraded data redundancy: 30/627 objects degraded (4.785%), 9 pgs degraded (PG_DEGRADED) 2024-05-06T13:59:49.030 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:48 smithi170 ceph-mon[65866]: pgmap v71: 161 pgs: 9 active+undersized+degraded, 12 active+undersized, 140 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 30/627 objects degraded (4.785%) 2024-05-06T13:59:49.030 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:48 smithi170 ceph-mon[65866]: Health check update: Degraded data redundancy: 30/627 objects degraded (4.785%), 9 pgs degraded (PG_DEGRADED) 2024-05-06T13:59:49.302 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:59:49 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T13:59:49.059+0000 7f5cd1795700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-05-06T13:59:50.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:49 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:50.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:49 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:50.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:49 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:59:50.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:49 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:59:50.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:49 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:50.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:49 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:59:50.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:49 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:59:50.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:49 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:59:50.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:49 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:59:50.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:49 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-06T13:59:50.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:49 smithi077 ceph-mon[84247]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-06T13:59:50.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:49 smithi077 ceph-mon[84247]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-05-06T13:59:50.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:49 smithi077 ceph-mon[84247]: pgmap v72: 161 pgs: 9 active+undersized+degraded, 12 active+undersized, 140 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 30/627 objects degraded (4.785%) 2024-05-06T13:59:50.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:49 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:50.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:49 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:50.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:49 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:59:50.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:49 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:59:50.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:49 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:50.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:49 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:59:50.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:49 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:59:50.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:49 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:59:50.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:49 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:59:50.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:49 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-06T13:59:50.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:49 smithi077 ceph-mon[89212]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-06T13:59:50.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:49 smithi077 ceph-mon[89212]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-05-06T13:59:50.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:49 smithi077 ceph-mon[89212]: pgmap v72: 161 pgs: 9 active+undersized+degraded, 12 active+undersized, 140 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 30/627 objects degraded (4.785%) 2024-05-06T13:59:50.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:49 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:50.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:49 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:50.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:49 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T13:59:50.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:49 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T13:59:50.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:49 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:50.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:49 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T13:59:50.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:49 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:59:50.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:49 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:59:50.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:49 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T13:59:50.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:49 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-06T13:59:50.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:49 smithi170 ceph-mon[65866]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-06T13:59:50.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:49 smithi170 ceph-mon[65866]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-05-06T13:59:50.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:49 smithi170 ceph-mon[65866]: pgmap v72: 161 pgs: 9 active+undersized+degraded, 12 active+undersized, 140 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 30/627 objects degraded (4.785%) 2024-05-06T13:59:51.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:51 smithi077 ceph-mon[84247]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-05-06T13:59:51.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:51 smithi077 ceph-mon[84247]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 30/627 objects degraded (4.785%), 9 pgs degraded) 2024-05-06T13:59:51.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:51 smithi077 ceph-mon[84247]: Cluster is now healthy 2024-05-06T13:59:51.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:51 smithi077 ceph-mon[89212]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-05-06T13:59:51.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:51 smithi077 ceph-mon[89212]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 30/627 objects degraded (4.785%), 9 pgs degraded) 2024-05-06T13:59:51.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:51 smithi077 ceph-mon[89212]: Cluster is now healthy 2024-05-06T13:59:51.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:51 smithi170 ceph-mon[65866]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-05-06T13:59:51.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:51 smithi170 ceph-mon[65866]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 30/627 objects degraded (4.785%), 9 pgs degraded) 2024-05-06T13:59:51.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:51 smithi170 ceph-mon[65866]: Cluster is now healthy 2024-05-06T13:59:54.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 13:59:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:13:59:53] "GET /metrics HTTP/1.1" 200 38147 "" "Prometheus/2.43.0" 2024-05-06T13:59:54.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:54 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:59:54.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:54 smithi077 ceph-mon[84247]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:54.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:54 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:59:54.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:54 smithi077 ceph-mon[89212]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:54.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:54 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T13:59:54.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:54 smithi170 ceph-mon[65866]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T13:59:56.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:56 smithi077 ceph-mon[84247]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T13:59:56.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:56 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:56.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:56 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:59:56.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:56 smithi077 ceph-mon[89212]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T13:59:56.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:56 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:56.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:56 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:59:56.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:56 smithi170 ceph-mon[65866]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T13:59:56.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:56 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:56.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:56 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T13:59:57.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:57 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:57.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 13:59:57 smithi077 ceph-mon[84247]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-05-06T13:59:57.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:57 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:57.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 13:59:57 smithi077 ceph-mon[89212]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-05-06T13:59:57.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:57 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T13:59:57.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 13:59:57 smithi170 ceph-mon[65866]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-05-06T14:00:00.353 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T14:00:00.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:00 smithi077 ceph-mon[89212]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:00.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:00 smithi077 ceph-mon[89212]: overall HEALTH_OK 2024-05-06T14:00:00.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:00 smithi077 ceph-mon[84247]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:00.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:00 smithi077 ceph-mon[84247]: overall HEALTH_OK 2024-05-06T14:00:00.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:00 smithi170 ceph-mon[65866]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:00.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:00 smithi170 ceph-mon[65866]: overall HEALTH_OK 2024-05-06T14:00:00.999 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T14:00:01.000 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (4m) 16s ago 27m 18.1M - 0.25.0 c8568f914cd2 997175e3e64e 2024-05-06T14:00:01.000 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (4m) 2m ago 26m 80.6M - 9.4.12 2bacad6d85d8 140bb9a2760a 2024-05-06T14:00:01.000 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (5m) 16s ago 26m 59.9M - 3.5 e1d6a67b021e 9353a303fb55 2024-05-06T14:00:01.000 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283,8765 running (6m) 2m ago 31m 447M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 125521be53a3 2024-05-06T14:00:01.000 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (15m) 16s ago 33m 503M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T14:00:01.000 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (2m) 16s ago 33m 45.1M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 5e50731ccbe8 2024-05-06T14:00:01.000 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (3m) 2m ago 31m 36.3M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 61cd48b3d0ff 2024-05-06T14:00:01.000 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (2m) 16s ago 32m 36.2M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 8a473cc2ffd1 2024-05-06T14:00:01.000 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (5m) 16s ago 27m 16.5M - 1.5.0 0da6a335fe13 67f45fa4968b 2024-05-06T14:00:01.000 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (5m) 2m ago 27m 16.4M - 1.5.0 0da6a335fe13 02081b71a5dd 2024-05-06T14:00:01.000 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (21s) 16s ago 31m 11.7M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 fbe876a933ed 2024-05-06T14:00:01.000 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (31m) 16s ago 30m 58.3M 2992M 17.2.0 e1d6a67b021e 1d9e6609b585 2024-05-06T14:00:01.000 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (30m) 16s ago 30m 53.2M 2992M 17.2.0 e1d6a67b021e 37a61d295111 2024-05-06T14:00:01.000 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (30m) 16s ago 30m 53.6M 2992M 17.2.0 e1d6a67b021e 8feb7c077aaa 2024-05-06T14:00:01.000 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (29m) 2m ago 29m 58.5M 3248M 17.2.0 e1d6a67b021e f865acc44a06 2024-05-06T14:00:01.001 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (29m) 2m ago 29m 52.8M 3248M 17.2.0 e1d6a67b021e a55bd4122f6f 2024-05-06T14:00:01.001 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (29m) 2m ago 29m 51.4M 3248M 17.2.0 e1d6a67b021e 362938a0b357 2024-05-06T14:00:01.001 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (28m) 2m ago 28m 54.1M 3248M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T14:00:01.001 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (5m) 2m ago 28m 60.1M - 2.43.0 a07b618ecd1d 10d5c513bb9b 2024-05-06T14:00:01.001 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (26m) 16s ago 26m 78.8M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T14:00:01.001 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (26m) 2m ago 26m 77.2M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T14:00:01.394 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:00:01.394 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-05-06T14:00:01.395 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T14:00:01.395 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:00:01.395 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-05-06T14:00:01.395 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T14:00:01.395 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:00:01.395 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-05-06T14:00:01.395 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-05-06T14:00:01.395 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 1 2024-05-06T14:00:01.395 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:00:01.395 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw": { 2024-05-06T14:00:01.395 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T14:00:01.395 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:00:01.395 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-05-06T14:00:01.395 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-05-06T14:00:01.395 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 6 2024-05-06T14:00:01.396 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T14:00:01.396 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:00:01.725 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:00:01.725 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T14:00:01.725 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-05-06T14:00:01.725 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 2 daemons (1 remaining).", 2024-05-06T14:00:01.725 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [], 2024-05-06T14:00:01.725 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "1/8 daemons upgraded", 2024-05-06T14:00:01.725 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "Currently upgrading osd daemons", 2024-05-06T14:00:01.725 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-05-06T14:00:01.725 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:00:02.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:02 smithi077 ceph-mon[84247]: from='client.54282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:02.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:02 smithi077 ceph-mon[84247]: from='client.64222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:02.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:02 smithi077 ceph-mon[84247]: from='client.64228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:02.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:02 smithi077 ceph-mon[84247]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:02.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:02 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/1368289034' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:02.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:02 smithi077 ceph-mon[89212]: from='client.54282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:02.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:02 smithi077 ceph-mon[89212]: from='client.64222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:02.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:02 smithi077 ceph-mon[89212]: from='client.64228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:02.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:02 smithi077 ceph-mon[89212]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:02.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:02 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/1368289034' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:02.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:02 smithi170 ceph-mon[65866]: from='client.54282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:02.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:02 smithi170 ceph-mon[65866]: from='client.64222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:02.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:02 smithi170 ceph-mon[65866]: from='client.64228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:02.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:02 smithi170 ceph-mon[65866]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:02.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:02 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/1368289034' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:03.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:03 smithi077 ceph-mon[84247]: from='client.64240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:03.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:03 smithi077 ceph-mon[89212]: from='client.64240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:03.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:03 smithi170 ceph-mon[65866]: from='client.64240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:04.053 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:00:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:00:03] "GET /metrics HTTP/1.1" 200 38169 "" "Prometheus/2.43.0" 2024-05-06T14:00:04.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:04 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:00:04.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:04 smithi077 ceph-mon[84247]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:00:04.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:04 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-06T14:00:04.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:04 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:00:04.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:04 smithi077 ceph-mon[89212]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:00:04.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:04 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-06T14:00:04.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:04 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:00:04.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:04 smithi170 ceph-mon[65866]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:00:04.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:04 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-06T14:00:05.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:05 smithi077 ceph-mon[84247]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-06T14:00:05.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:05 smithi077 ceph-mon[84247]: Upgrade: osd.1 is safe to restart 2024-05-06T14:00:05.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:05 smithi077 ceph-mon[89212]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-06T14:00:05.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:05 smithi077 ceph-mon[89212]: Upgrade: osd.1 is safe to restart 2024-05-06T14:00:05.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:05 smithi170 ceph-mon[65866]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-05-06T14:00:05.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:05 smithi170 ceph-mon[65866]: Upgrade: osd.1 is safe to restart 2024-05-06T14:00:06.491 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:06 smithi077 ceph-mon[89212]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:06.491 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:06 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:06.491 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:06 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-06T14:00:06.491 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:06 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:00:06.491 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:06 smithi077 ceph-mon[84247]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:06.492 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:06 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:06.492 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:06 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-06T14:00:06.492 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:06 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:00:06.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:06 smithi170 ceph-mon[65866]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:06.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:06 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:06.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:06 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-06T14:00:06.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:06 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:00:07.281 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:07 smithi077 ceph-mon[84247]: Upgrade: Updating osd.1 2024-05-06T14:00:07.281 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:07 smithi077 ceph-mon[84247]: Deploying daemon osd.1 on smithi077 2024-05-06T14:00:07.281 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:07 smithi077 ceph-mon[89212]: Upgrade: Updating osd.1 2024-05-06T14:00:07.281 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:07 smithi077 ceph-mon[89212]: Deploying daemon osd.1 on smithi077 2024-05-06T14:00:07.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:07 smithi170 ceph-mon[65866]: Upgrade: Updating osd.1 2024-05-06T14:00:07.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:07 smithi170 ceph-mon[65866]: Deploying daemon osd.1 on smithi077 2024-05-06T14:00:08.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:08 smithi077 ceph-mon[84247]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:00:08.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:08 smithi077 ceph-mon[89212]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:00:08.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:08 smithi170 ceph-mon[65866]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:00:09.367 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:09 smithi077 systemd[1]: Stopping Ceph osd.1 for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:00:09.802 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:09 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1[32976]: 2024-05-06T14:00:09.365+0000 7fd505bae700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:00:09.803 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:09 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1[32976]: 2024-05-06T14:00:09.365+0000 7fd505bae700 -1 osd.1 118 *** Got signal Terminated *** 2024-05-06T14:00:09.803 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:09 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1[32976]: 2024-05-06T14:00:09.365+0000 7fd505bae700 -1 osd.1 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T14:00:10.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:10 smithi077 ceph-mon[84247]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:10.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:10 smithi077 ceph-mon[84247]: osd.1 marked itself down and dead 2024-05-06T14:00:10.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:10 smithi077 ceph-mon[89212]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:10.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:10 smithi077 ceph-mon[89212]: osd.1 marked itself down and dead 2024-05-06T14:00:10.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:10 smithi170 ceph-mon[65866]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:10.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:10 smithi170 ceph-mon[65866]: osd.1 marked itself down and dead 2024-05-06T14:00:10.918 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:10 smithi077 podman[101413]: 2024-05-06 14:00:10.635823197 +0000 UTC m=+1.418520224 container died 1d9e6609b585bba945e7d1325ad2b89ac5c9e15e1183384503aafb959410202b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, name=centos-stream, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, vcs-type=git, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, distribution-scope=public, version=8, com.redhat.component=centos-stream-container, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-05-06T14:00:11.201 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:11 smithi077 podman[101413]: 2024-05-06 14:00:11.116204062 +0000 UTC m=+1.898901132 container cleanup 1d9e6609b585bba945e7d1325ad2b89ac5c9e15e1183384503aafb959410202b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., vcs-type=git, name=centos-stream, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, version=8, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, release=754, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-05-06T14:00:11.201 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:11 smithi077 bash[101413]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1 2024-05-06T14:00:11.478 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:11 smithi170 ceph-mon[65866]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T14:00:11.478 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:11 smithi170 ceph-mon[65866]: osdmap e119: 8 total, 7 up, 8 in 2024-05-06T14:00:11.478 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:11 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:11.478 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:11 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:00:11.478 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:11 smithi170 ceph-mon[65866]: osdmap e120: 8 total, 7 up, 8 in 2024-05-06T14:00:11.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:11 smithi077 ceph-mon[89212]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T14:00:11.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:11 smithi077 ceph-mon[89212]: osdmap e119: 8 total, 7 up, 8 in 2024-05-06T14:00:11.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:11 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:11.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:11 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:00:11.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:11 smithi077 ceph-mon[89212]: osdmap e120: 8 total, 7 up, 8 in 2024-05-06T14:00:11.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:11 smithi077 ceph-mon[84247]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T14:00:11.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:11 smithi077 ceph-mon[84247]: osdmap e119: 8 total, 7 up, 8 in 2024-05-06T14:00:11.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:11 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:11.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:11 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:00:11.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:11 smithi077 ceph-mon[84247]: osdmap e120: 8 total, 7 up, 8 in 2024-05-06T14:00:11.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:11 smithi077 podman[101430]: 2024-05-06 14:00:11.409853283 +0000 UTC m=+0.771338136 container remove 1d9e6609b585bba945e7d1325ad2b89ac5c9e15e1183384503aafb959410202b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux , distribution-scope=public, io.openshift.expose-services=, io.buildah.version=1.19.8, version=8, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754) 2024-05-06T14:00:12.030 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:11 smithi077 podman[101524]: 2024-05-06 14:00:11.653838661 +0000 UTC m=+0.018260448 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:00:12.030 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:11 smithi077 podman[101524]: 2024-05-06 14:00:11.761029485 +0000 UTC m=+0.125451287 container create 830f7ed78585cd5a66681508ddfbca5f1475307959ba0b571dae01dcc85c7d93 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=) 2024-05-06T14:00:12.030 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:11 smithi077 podman[101524]: 2024-05-06 14:00:11.954944476 +0000 UTC m=+0.319366236 container init 830f7ed78585cd5a66681508ddfbca5f1475307959ba0b571dae01dcc85c7d93 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-70ce953, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True) 2024-05-06T14:00:12.030 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:11 smithi077 podman[101524]: 2024-05-06 14:00:11.958218234 +0000 UTC m=+0.322640002 container start 830f7ed78585cd5a66681508ddfbca5f1475307959ba0b571dae01dcc85c7d93 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-06T14:00:12.031 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:12 smithi077 podman[101524]: 2024-05-06 14:00:12.029887951 +0000 UTC m=+0.394309723 container attach 830f7ed78585cd5a66681508ddfbca5f1475307959ba0b571dae01dcc85c7d93 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-70ce953, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-06T14:00:12.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:12 smithi077 ceph-mon[84247]: pgmap v84: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T14:00:12.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:12 smithi077 ceph-mon[89212]: pgmap v84: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T14:00:12.303 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:12 smithi077 podman[101524]: 2024-05-06 14:00:12.257920969 +0000 UTC m=+0.622342735 container died 830f7ed78585cd5a66681508ddfbca5f1475307959ba0b571dae01dcc85c7d93 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-06T14:00:12.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:12 smithi170 ceph-mon[65866]: pgmap v84: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T14:00:13.209 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:12 smithi077 podman[101549]: 2024-05-06 14:00:12.833328339 +0000 UTC m=+0.570081562 container remove 830f7ed78585cd5a66681508ddfbca5f1475307959ba0b571dae01dcc85c7d93 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-70ce953, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-05-06T14:00:13.210 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:12 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.1.service: Deactivated successfully. 2024-05-06T14:00:13.210 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:12 smithi077 systemd[1]: Stopped Ceph osd.1 for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T14:00:13.210 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:12 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.1.service: Consumed 15.281s CPU time. 2024-05-06T14:00:13.210 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:13 smithi077 systemd[1]: Starting Ceph osd.1 for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:00:13.537 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:13 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:00:13.537 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:13 smithi077 ceph-mon[84247]: pgmap v86: 161 pgs: 19 active+undersized+degraded, 1 stale+active+clean, 37 active+undersized, 104 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-05-06T14:00:13.537 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:13 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:00:13.538 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:13 smithi077 ceph-mon[89212]: pgmap v86: 161 pgs: 19 active+undersized+degraded, 1 stale+active+clean, 37 active+undersized, 104 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-05-06T14:00:13.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:13 smithi077 podman[101654]: 2024-05-06 14:00:13.1905709 +0000 UTC m=+0.017991761 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:00:13.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:13 smithi077 podman[101654]: 2024-05-06 14:00:13.318319788 +0000 UTC m=+0.145740641 container create 59ae8017312884800e3c4843f32ed9dc2ac32cf59dbf20a0dc759758a5344e68 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1-activate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-06T14:00:13.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:13 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:00:13.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:13 smithi170 ceph-mon[65866]: pgmap v86: 161 pgs: 19 active+undersized+degraded, 1 stale+active+clean, 37 active+undersized, 104 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-05-06T14:00:13.802 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:00:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:00:13] "GET /metrics HTTP/1.1" 200 38166 "" "Prometheus/2.43.0" 2024-05-06T14:00:13.803 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:13 smithi077 podman[101654]: 2024-05-06 14:00:13.537325502 +0000 UTC m=+0.364746349 container init 59ae8017312884800e3c4843f32ed9dc2ac32cf59dbf20a0dc759758a5344e68 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1-activate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-05-06T14:00:13.803 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:13 smithi077 podman[101654]: 2024-05-06 14:00:13.540640833 +0000 UTC m=+0.368061682 container start 59ae8017312884800e3c4843f32ed9dc2ac32cf59dbf20a0dc759758a5344e68 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1-activate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-05-06T14:00:13.803 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:13 smithi077 podman[101654]: 2024-05-06 14:00:13.670677352 +0000 UTC m=+0.498098209 container attach 59ae8017312884800e3c4843f32ed9dc2ac32cf59dbf20a0dc759758a5344e68 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-70ce953, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=) 2024-05-06T14:00:14.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:14 smithi077 ceph-mon[84247]: Health check failed: Degraded data redundancy: 77/627 objects degraded (12.281%), 19 pgs degraded (PG_DEGRADED) 2024-05-06T14:00:14.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:14 smithi077 ceph-mon[89212]: Health check failed: Degraded data redundancy: 77/627 objects degraded (12.281%), 19 pgs degraded (PG_DEGRADED) 2024-05-06T14:00:14.303 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1-activate[101669]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-06T14:00:14.303 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1-activate[101669]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-05-06T14:00:14.304 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:13 smithi077 bash[101654]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-06T14:00:14.304 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:13 smithi077 bash[101654]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-05-06T14:00:14.304 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:13 smithi077 bash[101654]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-05-06T14:00:14.304 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:13 smithi077 bash[101654]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-05-06T14:00:14.304 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:13 smithi077 bash[101654]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-05-06T14:00:14.304 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:13 smithi077 bash[101654]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-06T14:00:14.304 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:13 smithi077 bash[101654]: --> ceph-volume raw activate successful for osd ID: 1 2024-05-06T14:00:14.304 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1-activate[101669]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-05-06T14:00:14.304 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1-activate[101669]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-05-06T14:00:14.304 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1-activate[101669]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-05-06T14:00:14.305 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1-activate[101669]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-06T14:00:14.305 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1-activate[101669]: --> ceph-volume raw activate successful for osd ID: 1 2024-05-06T14:00:14.305 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:14 smithi077 podman[101654]: 2024-05-06 14:00:14.004416828 +0000 UTC m=+0.831837666 container died 59ae8017312884800e3c4843f32ed9dc2ac32cf59dbf20a0dc759758a5344e68 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1-activate, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-70ce953) 2024-05-06T14:00:14.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:14 smithi170 ceph-mon[65866]: Health check failed: Degraded data redundancy: 77/627 objects degraded (12.281%), 19 pgs degraded (PG_DEGRADED) 2024-05-06T14:00:14.942 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:14 smithi077 podman[101789]: 2024-05-06 14:00:14.616113021 +0000 UTC m=+0.606593011 container remove 59ae8017312884800e3c4843f32ed9dc2ac32cf59dbf20a0dc759758a5344e68 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1-activate, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-70ce953, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T14:00:14.942 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:14 smithi077 podman[101856]: 2024-05-06 14:00:14.733443334 +0000 UTC m=+0.017364869 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:00:14.942 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:14 smithi077 podman[101856]: 2024-05-06 14:00:14.942095338 +0000 UTC m=+0.226016865 container create 6f664e567c93b2d2ed95c3440466046cfeeaacc7282b685296ace3e8528a17a2 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0) 2024-05-06T14:00:15.201 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:15 smithi077 podman[101856]: 2024-05-06 14:00:15.178362669 +0000 UTC m=+0.462284195 container init 6f664e567c93b2d2ed95c3440466046cfeeaacc7282b685296ace3e8528a17a2 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-05-06T14:00:15.202 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:15 smithi077 podman[101856]: 2024-05-06 14:00:15.181360158 +0000 UTC m=+0.465281686 container start 6f664e567c93b2d2ed95c3440466046cfeeaacc7282b685296ace3e8528a17a2 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T14:00:15.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:15 smithi077 ceph-mon[84247]: pgmap v87: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-05-06T14:00:15.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:15 smithi077 ceph-mon[89212]: pgmap v87: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-05-06T14:00:15.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:15 smithi077 ceph-osd[101875]: -- 172.21.15.77:0/1674252140 <== mon.1 v2:172.21.15.77:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a46815f680 con 0x55a468129400 2024-05-06T14:00:15.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:15 smithi077 bash[101856]: 6f664e567c93b2d2ed95c3440466046cfeeaacc7282b685296ace3e8528a17a2 2024-05-06T14:00:15.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:15 smithi077 systemd[1]: Started Ceph osd.1 for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T14:00:15.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:15 smithi170 ceph-mon[65866]: pgmap v87: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-05-06T14:00:16.052 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:15 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1[101871]: 2024-05-06T14:00:15.724+0000 7fb6cfd5a7c0 -1 Falling back to public interface 2024-05-06T14:00:16.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:16.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:16.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:16.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:16.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:16.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:16.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:16.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:16.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:17.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:17 smithi170 ceph-mon[65866]: pgmap v88: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-05-06T14:00:17.746 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:17 smithi077 ceph-mon[84247]: pgmap v88: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-05-06T14:00:17.746 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:17 smithi077 ceph-mon[89212]: pgmap v88: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-05-06T14:00:18.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:18 smithi077 ceph-mon[84247]: from='osd.1 [v2:172.21.15.77:6810/2510866306,v1:172.21.15.77:6811/2510866306]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-06T14:00:18.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:18 smithi077 ceph-mon[89212]: from='osd.1 [v2:172.21.15.77:6810/2510866306,v1:172.21.15.77:6811/2510866306]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-06T14:00:18.803 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:18 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1[101871]: 2024-05-06T14:00:18.466+0000 7fb6cfd5a7c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-05-06T14:00:18.803 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:18 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1[101871]: 2024-05-06T14:00:18.496+0000 7fb6cfd5a7c0 -1 osd.1 118 log_to_monitors true 2024-05-06T14:00:18.803 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:00:18 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1[101871]: 2024-05-06T14:00:18.586+0000 7fb6c11b1700 -1 osd.1 118 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T14:00:18.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:18 smithi170 ceph-mon[65866]: from='osd.1 [v2:172.21.15.77:6810/2510866306,v1:172.21.15.77:6811/2510866306]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-06T14:00:19.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:19 smithi170 ceph-mon[65866]: from='osd.1 [v2:172.21.15.77:6810/2510866306,v1:172.21.15.77:6811/2510866306]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-06T14:00:19.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:19 smithi170 ceph-mon[65866]: osdmap e121: 8 total, 7 up, 8 in 2024-05-06T14:00:19.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:19 smithi170 ceph-mon[65866]: from='osd.1 [v2:172.21.15.77:6810/2510866306,v1:172.21.15.77:6811/2510866306]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T14:00:19.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:19 smithi170 ceph-mon[65866]: pgmap v90: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-05-06T14:00:20.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:19 smithi077 ceph-mon[84247]: from='osd.1 [v2:172.21.15.77:6810/2510866306,v1:172.21.15.77:6811/2510866306]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-06T14:00:20.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:19 smithi077 ceph-mon[84247]: osdmap e121: 8 total, 7 up, 8 in 2024-05-06T14:00:20.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:19 smithi077 ceph-mon[84247]: from='osd.1 [v2:172.21.15.77:6810/2510866306,v1:172.21.15.77:6811/2510866306]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T14:00:20.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:19 smithi077 ceph-mon[84247]: pgmap v90: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-05-06T14:00:20.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:19 smithi077 ceph-mon[89212]: from='osd.1 [v2:172.21.15.77:6810/2510866306,v1:172.21.15.77:6811/2510866306]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-06T14:00:20.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:19 smithi077 ceph-mon[89212]: osdmap e121: 8 total, 7 up, 8 in 2024-05-06T14:00:20.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:19 smithi077 ceph-mon[89212]: from='osd.1 [v2:172.21.15.77:6810/2510866306,v1:172.21.15.77:6811/2510866306]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T14:00:20.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:19 smithi077 ceph-mon[89212]: pgmap v90: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-05-06T14:00:21.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:20 smithi077 ceph-mon[84247]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T14:00:21.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:20 smithi077 ceph-mon[84247]: osd.1 [v2:172.21.15.77:6810/2510866306,v1:172.21.15.77:6811/2510866306] boot 2024-05-06T14:00:21.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:20 smithi077 ceph-mon[84247]: osdmap e122: 8 total, 8 up, 8 in 2024-05-06T14:00:21.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:20 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T14:00:21.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:20 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:21.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:20 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:21.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:20 smithi077 ceph-mon[89212]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T14:00:21.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:20 smithi077 ceph-mon[89212]: osd.1 [v2:172.21.15.77:6810/2510866306,v1:172.21.15.77:6811/2510866306] boot 2024-05-06T14:00:21.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:20 smithi077 ceph-mon[89212]: osdmap e122: 8 total, 8 up, 8 in 2024-05-06T14:00:21.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:20 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T14:00:21.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:20 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:21.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:20 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:21.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:20 smithi170 ceph-mon[65866]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T14:00:21.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:20 smithi170 ceph-mon[65866]: osd.1 [v2:172.21.15.77:6810/2510866306,v1:172.21.15.77:6811/2510866306] boot 2024-05-06T14:00:21.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:20 smithi170 ceph-mon[65866]: osdmap e122: 8 total, 8 up, 8 in 2024-05-06T14:00:21.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:20 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-06T14:00:21.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:20 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:21.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:20 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:21.667 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:21 smithi077 ceph-mon[89212]: osdmap e123: 8 total, 8 up, 8 in 2024-05-06T14:00:21.667 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:21 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:21.667 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:21 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:21.667 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:21 smithi077 ceph-mon[89212]: pgmap v93: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 84/627 objects degraded (13.397%) 2024-05-06T14:00:21.668 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:21 smithi077 ceph-mon[84247]: osdmap e123: 8 total, 8 up, 8 in 2024-05-06T14:00:21.668 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:21 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:21.668 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:21 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:21.668 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:21 smithi077 ceph-mon[84247]: pgmap v93: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 84/627 objects degraded (13.397%) 2024-05-06T14:00:22.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:21 smithi170 ceph-mon[65866]: osdmap e123: 8 total, 8 up, 8 in 2024-05-06T14:00:22.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:21 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:22.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:21 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:22.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:21 smithi170 ceph-mon[65866]: pgmap v93: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 84/627 objects degraded (13.397%) 2024-05-06T14:00:22.900 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:22 smithi077 ceph-mon[84247]: Health check update: Degraded data redundancy: 84/627 objects degraded (13.397%), 21 pgs degraded (PG_DEGRADED) 2024-05-06T14:00:22.901 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:22 smithi077 ceph-mon[89212]: Health check update: Degraded data redundancy: 84/627 objects degraded (13.397%), 21 pgs degraded (PG_DEGRADED) 2024-05-06T14:00:23.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:22 smithi170 ceph-mon[65866]: Health check update: Degraded data redundancy: 84/627 objects degraded (13.397%), 21 pgs degraded (PG_DEGRADED) 2024-05-06T14:00:23.777 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:00:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:00:23] "GET /metrics HTTP/1.1" 200 38166 "" "Prometheus/2.43.0" 2024-05-06T14:00:24.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:23 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:00:24.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:23 smithi077 ceph-mon[89212]: pgmap v94: 161 pgs: 8 active+undersized+degraded, 19 active+undersized, 134 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 37/627 objects degraded (5.901%) 2024-05-06T14:00:24.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:23 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:00:24.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:23 smithi077 ceph-mon[84247]: pgmap v94: 161 pgs: 8 active+undersized+degraded, 19 active+undersized, 134 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 37/627 objects degraded (5.901%) 2024-05-06T14:00:24.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:23 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:00:24.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:23 smithi170 ceph-mon[65866]: pgmap v94: 161 pgs: 8 active+undersized+degraded, 19 active+undersized, 134 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 37/627 objects degraded (5.901%) 2024-05-06T14:00:25.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:25.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:25.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:00:25.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:00:25.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:25.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:00:25.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:25.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:25.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all nvmeof 2024-05-06T14:00:25.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:25.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:25.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:25.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:25.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:25.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:25.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:25.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: Upgrade: Finalizing container_image settings 2024-05-06T14:00:25.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-05-06T14:00:25.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-05-06T14:00:25.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:00:25.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-05-06T14:00:25.555 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-05-06T14:00:25.555 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-05-06T14:00:25.555 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-05-06T14:00:25.555 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-05-06T14:00:25.555 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-05-06T14:00:25.555 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-05-06T14:00:25.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:25.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:25.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:00:25.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:00:25.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:25.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:00:25.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:25.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:25.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all nvmeof 2024-05-06T14:00:25.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:25.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:25.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:25.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:25.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:25.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:25.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:25.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: Upgrade: Finalizing container_image settings 2024-05-06T14:00:25.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-05-06T14:00:25.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-05-06T14:00:25.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:00:25.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-05-06T14:00:25.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-05-06T14:00:25.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-05-06T14:00:25.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-05-06T14:00:25.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-05-06T14:00:25.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-05-06T14:00:25.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-05-06T14:00:25.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:00:25.558 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-05-06T14:00:25.558 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-05-06T14:00:25.558 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-05-06T14:00:25.558 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-05-06T14:00:25.558 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-05-06T14:00:25.558 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:00:25.558 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:00:25.558 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:00:25.558 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:00:25.559 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:00:25.559 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:00:25.559 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: Upgrade: Complete! 2024-05-06T14:00:25.559 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-05-06T14:00:25.559 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-05-06T14:00:25.559 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:00:25.559 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:00:25.559 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:25.559 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:00:25.559 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:00:25.559 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:25.559 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[89212]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 781 B/s rd, 0 op/s 2024-05-06T14:00:25.560 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:00:25.560 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-05-06T14:00:25.560 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-05-06T14:00:25.560 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-05-06T14:00:25.560 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-05-06T14:00:25.560 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-05-06T14:00:25.560 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:00:25.560 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:00:25.560 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:00:25.560 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:00:25.560 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:00:25.560 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:00:25.560 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: Upgrade: Complete! 2024-05-06T14:00:25.560 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-05-06T14:00:25.560 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-05-06T14:00:25.561 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:00:25.561 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:00:25.561 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:25.561 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:00:25.561 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:00:25.561 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:25.561 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:25 smithi077 ceph-mon[84247]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 781 B/s rd, 0 op/s 2024-05-06T14:00:25.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:25.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:25.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:00:25.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:00:25.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:25.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:00:25.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:25.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:25.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all nvmeof 2024-05-06T14:00:25.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:25.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:25.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:25.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:25.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:25.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:25.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:25.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: Upgrade: Finalizing container_image settings 2024-05-06T14:00:25.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-05-06T14:00:25.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-05-06T14:00:25.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:00:25.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-05-06T14:00:25.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-05-06T14:00:25.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-05-06T14:00:25.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-05-06T14:00:25.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-05-06T14:00:25.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-05-06T14:00:25.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-05-06T14:00:25.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:00:25.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-05-06T14:00:25.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-05-06T14:00:25.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-05-06T14:00:25.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-05-06T14:00:25.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-05-06T14:00:25.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:00:25.580 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:00:25.580 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:00:25.580 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:00:25.580 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:00:25.580 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:00:25.580 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: Upgrade: Complete! 2024-05-06T14:00:25.580 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-05-06T14:00:25.580 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-05-06T14:00:25.580 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:00:25.580 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:00:25.580 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:25.580 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:00:25.580 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:00:25.580 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:25.580 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:25 smithi170 ceph-mon[65866]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 781 B/s rd, 0 op/s 2024-05-06T14:00:26.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:26 smithi077 ceph-mon[84247]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 37/627 objects degraded (5.901%), 8 pgs degraded) 2024-05-06T14:00:26.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:26 smithi077 ceph-mon[84247]: Cluster is now healthy 2024-05-06T14:00:26.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:26 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:26.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:26 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:00:26.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:26 smithi077 ceph-mon[89212]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 37/627 objects degraded (5.901%), 8 pgs degraded) 2024-05-06T14:00:26.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:26 smithi077 ceph-mon[89212]: Cluster is now healthy 2024-05-06T14:00:26.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:26 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:26.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:26 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:00:26.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:26 smithi170 ceph-mon[65866]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 37/627 objects degraded (5.901%), 8 pgs degraded) 2024-05-06T14:00:26.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:26 smithi170 ceph-mon[65866]: Cluster is now healthy 2024-05-06T14:00:26.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:26 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:26.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:26 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:00:27.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:27 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:27.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:27 smithi077 ceph-mon[89212]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:00:27.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:27 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:27.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:27 smithi077 ceph-mon[84247]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:00:27.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:27 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:27.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:27 smithi170 ceph-mon[65866]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:00:30.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:30 smithi077 ceph-mon[84247]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T14:00:30.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:30 smithi077 ceph-mon[89212]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T14:00:30.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:30 smithi170 ceph-mon[65866]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T14:00:32.354 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:32 smithi077 ceph-mon[84247]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 971 B/s rd, 0 op/s 2024-05-06T14:00:32.355 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:32.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:32 smithi077 ceph-mon[89212]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 971 B/s rd, 0 op/s 2024-05-06T14:00:32.355 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:32.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:32 smithi170 ceph-mon[65866]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 971 B/s rd, 0 op/s 2024-05-06T14:00:32.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:32.666 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch ps' 2024-05-06T14:00:33.220 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:33 smithi077 ceph-mon[84247]: from='client.64252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:33.221 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:33 smithi077 ceph-mon[89212]: from='client.64252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:33.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:33 smithi170 ceph-mon[65866]: from='client.64252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:33.915 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:00:33 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:00:33] "GET /metrics HTTP/1.1" 200 38175 "" "Prometheus/2.43.0" 2024-05-06T14:00:34.435 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:34 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:00:34.435 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:34 smithi077 ceph-mon[84247]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:00:34.435 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:34 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:00:34.435 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:34 smithi077 ceph-mon[89212]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:00:34.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:34 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:00:34.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:34 smithi170 ceph-mon[65866]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:00:34.736 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T14:00:34.736 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (5m) 15s ago 27m 26.6M - 0.25.0 c8568f914cd2 997175e3e64e 2024-05-06T14:00:34.736 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (4m) 2m ago 27m 80.6M - 9.4.12 2bacad6d85d8 140bb9a2760a 2024-05-06T14:00:34.736 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (6m) 15s ago 26m 61.9M - 3.5 e1d6a67b021e 9353a303fb55 2024-05-06T14:00:34.736 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283,8765 running (6m) 2m ago 32m 447M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 125521be53a3 2024-05-06T14:00:34.736 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (16m) 15s ago 33m 505M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T14:00:34.736 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (3m) 15s ago 33m 46.5M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 5e50731ccbe8 2024-05-06T14:00:34.736 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (3m) 2m ago 32m 36.3M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 61cd48b3d0ff 2024-05-06T14:00:34.736 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (2m) 15s ago 32m 37.7M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 8a473cc2ffd1 2024-05-06T14:00:34.736 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (6m) 15s ago 28m 16.0M - 1.5.0 0da6a335fe13 67f45fa4968b 2024-05-06T14:00:34.737 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (6m) 2m ago 28m 16.4M - 1.5.0 0da6a335fe13 02081b71a5dd 2024-05-06T14:00:34.737 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (55s) 15s ago 31m 41.7M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 fbe876a933ed 2024-05-06T14:00:34.737 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (19s) 15s ago 31m 12.6M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 6f664e567c93 2024-05-06T14:00:34.737 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (31m) 15s ago 31m 53.6M 2992M 17.2.0 e1d6a67b021e 37a61d295111 2024-05-06T14:00:34.737 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (30m) 15s ago 30m 54.1M 2992M 17.2.0 e1d6a67b021e 8feb7c077aaa 2024-05-06T14:00:34.737 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (30m) 2m ago 30m 58.5M 3248M 17.2.0 e1d6a67b021e f865acc44a06 2024-05-06T14:00:34.737 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (30m) 2m ago 30m 52.8M 3248M 17.2.0 e1d6a67b021e a55bd4122f6f 2024-05-06T14:00:34.737 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (29m) 2m ago 29m 51.4M 3248M 17.2.0 e1d6a67b021e 362938a0b357 2024-05-06T14:00:34.737 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (29m) 2m ago 29m 54.1M 3248M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T14:00:34.737 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (5m) 2m ago 28m 60.1M - 2.43.0 a07b618ecd1d 10d5c513bb9b 2024-05-06T14:00:34.737 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (26m) 15s ago 26m 78.9M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T14:00:34.737 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (26m) 2m ago 26m 77.2M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T14:00:35.382 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 2'"'"'' 2024-05-06T14:00:36.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:36 smithi077 ceph-mon[84247]: from='client.54324 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:36.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:36 smithi077 ceph-mon[84247]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:36.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:36 smithi077 ceph-mon[89212]: from='client.54324 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:36.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:36 smithi077 ceph-mon[89212]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:36.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:36 smithi170 ceph-mon[65866]: from='client.54324 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:36.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:36 smithi170 ceph-mon[65866]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:37.548 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T14:00:38.223 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 7'"'"'' 2024-05-06T14:00:38.525 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:38 smithi077 ceph-mon[84247]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:00:38.526 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:38 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/1183180681' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:38.526 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:38 smithi077 ceph-mon[89212]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:00:38.526 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:38 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/1183180681' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:38.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:38 smithi170 ceph-mon[65866]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:00:38.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:38 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/1183180681' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:40.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:40 smithi077 ceph-mon[84247]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:40.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:40 smithi077 ceph-mon[89212]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:40.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:40 smithi170 ceph-mon[65866]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:41.478 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:00:41.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:00:41.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:00:41.836 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T14:00:42.394 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:42 smithi077 ceph-mon[84247]: from='client.64270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:42.394 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:42 smithi077 ceph-mon[84247]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:42.394 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:42 smithi077 ceph-mon[89212]: from='client.64270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:42.394 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:42 smithi077 ceph-mon[89212]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:42.416 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch upgrade status' 2024-05-06T14:00:42.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:42 smithi170 ceph-mon[65866]: from='client.64270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:42.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:42 smithi170 ceph-mon[65866]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:43.857 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:00:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:00:43] "GET /metrics HTTP/1.1" 200 38165 "" "Prometheus/2.43.0" 2024-05-06T14:00:44.318 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:00:44.318 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": null, 2024-05-06T14:00:44.318 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": false, 2024-05-06T14:00:44.318 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "", 2024-05-06T14:00:44.318 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [], 2024-05-06T14:00:44.318 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": null, 2024-05-06T14:00:44.318 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "", 2024-05-06T14:00:44.318 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-05-06T14:00:44.318 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:00:44.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:44 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:00:44.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:44 smithi077 ceph-mon[84247]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:00:44.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:44 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:00:44.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:44 smithi077 ceph-mon[89212]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:00:44.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:44 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:00:44.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:44 smithi170 ceph-mon[65866]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:00:45.032 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph health detail' 2024-05-06T14:00:46.459 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:46 smithi077 ceph-mon[84247]: from='client.54342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:46.459 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:46 smithi077 ceph-mon[84247]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:46.459 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:46 smithi077 ceph-mon[89212]: from='client.54342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:46.459 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:46 smithi077 ceph-mon[89212]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:46.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:46 smithi170 ceph-mon[65866]: from='client.54342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:46.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:46 smithi170 ceph-mon[65866]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:47.059 INFO:teuthology.orchestra.run.smithi077.stdout:HEALTH_OK 2024-05-06T14:00:47.356 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:47 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/2423417098' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T14:00:47.356 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:47 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/2423417098' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T14:00:47.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:47 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/2423417098' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T14:00:47.790 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd --limit 1' 2024-05-06T14:00:48.344 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:48 smithi077 ceph-mon[84247]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:00:48.344 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:48 smithi077 ceph-mon[89212]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:00:48.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:48 smithi170 ceph-mon[65866]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:00:50.456 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:50 smithi077 ceph-mon[84247]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:50.456 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:50 smithi077 ceph-mon[89212]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:50.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:50 smithi170 ceph-mon[65866]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:51.453 INFO:teuthology.orchestra.run.smithi077.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:00:51.453 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:51 smithi077 ceph-mon[84247]: from='client.64288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:51.453 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:51 smithi077 ceph-mon[89212]: from='client.64288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:51.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:51 smithi170 ceph-mon[65866]: from='client.64288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:52.141 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-05-06T14:00:52.454 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:52 smithi077 ceph-mon[84247]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:52.455 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:52 smithi077 ceph-mon[84247]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:00:52.455 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:52 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:52.455 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:52 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:00:52.455 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:52 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:00:52.455 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:52 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:52.455 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:52 smithi077 ceph-mon[84247]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:00:52.455 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:52 smithi077 ceph-mon[89212]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:52.455 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:52 smithi077 ceph-mon[89212]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:00:52.455 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:52 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:52.455 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:52 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:00:52.455 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:52 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:00:52.456 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:52 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:52.456 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:52 smithi077 ceph-mon[89212]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:00:52.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:52 smithi170 ceph-mon[65866]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:52.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:52 smithi170 ceph-mon[65866]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:00:52.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:52 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:52.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:52 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:00:52.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:52 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:00:52.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:52 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:52.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:52 smithi170 ceph-mon[65866]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:00:53.380 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:53 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:00:53.380 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:53 smithi077 ceph-mon[84247]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:00:53.380 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:53 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:00:53.380 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:53 smithi077 ceph-mon[89212]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:00:53.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:53 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:00:53.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:53 smithi170 ceph-mon[65866]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:00:54.006 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:00:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:00:53] "GET /metrics HTTP/1.1" 200 38165 "" "Prometheus/2.43.0" 2024-05-06T14:00:54.108 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T14:00:54.262 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:54 smithi077 ceph-mon[84247]: from='client.54360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:54.262 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:54 smithi077 ceph-mon[89212]: from='client.54360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:54.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:54 smithi170 ceph-mon[65866]: from='client.54360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:54.774 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T14:00:54.774 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (5m) 35s ago 28m 26.6M - 0.25.0 c8568f914cd2 997175e3e64e 2024-05-06T14:00:54.775 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (5m) 3m ago 27m 80.6M - 9.4.12 2bacad6d85d8 140bb9a2760a 2024-05-06T14:00:54.775 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (6m) 35s ago 27m 61.9M - 3.5 e1d6a67b021e 9353a303fb55 2024-05-06T14:00:54.775 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283,8765 running (7m) 3m ago 32m 447M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 125521be53a3 2024-05-06T14:00:54.775 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (16m) 35s ago 34m 505M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T14:00:54.775 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (3m) 35s ago 34m 46.5M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 5e50731ccbe8 2024-05-06T14:00:54.775 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (4m) 3m ago 32m 36.3M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 61cd48b3d0ff 2024-05-06T14:00:54.775 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (2m) 35s ago 32m 37.7M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 8a473cc2ffd1 2024-05-06T14:00:54.775 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (6m) 35s ago 28m 16.0M - 1.5.0 0da6a335fe13 67f45fa4968b 2024-05-06T14:00:54.775 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (6m) 3m ago 28m 16.4M - 1.5.0 0da6a335fe13 02081b71a5dd 2024-05-06T14:00:54.775 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (75s) 35s ago 32m 41.7M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 fbe876a933ed 2024-05-06T14:00:54.775 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (40s) 35s ago 31m 12.6M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 6f664e567c93 2024-05-06T14:00:54.775 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (31m) 35s ago 31m 53.6M 2992M 17.2.0 e1d6a67b021e 37a61d295111 2024-05-06T14:00:54.775 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (31m) 35s ago 31m 54.1M 2992M 17.2.0 e1d6a67b021e 8feb7c077aaa 2024-05-06T14:00:54.775 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (30m) 3m ago 30m 58.5M 3248M 17.2.0 e1d6a67b021e f865acc44a06 2024-05-06T14:00:54.776 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (30m) 3m ago 30m 52.8M 3248M 17.2.0 e1d6a67b021e a55bd4122f6f 2024-05-06T14:00:54.776 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (29m) 3m ago 29m 51.4M 3248M 17.2.0 e1d6a67b021e 362938a0b357 2024-05-06T14:00:54.776 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (29m) 3m ago 29m 54.1M 3248M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T14:00:54.776 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (6m) 3m ago 28m 60.1M - 2.43.0 a07b618ecd1d 10d5c513bb9b 2024-05-06T14:00:54.776 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (27m) 35s ago 27m 78.9M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T14:00:54.776 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (27m) 3m ago 27m 77.2M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T14:00:55.202 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:00:55.202 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-05-06T14:00:55.202 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T14:00:55.202 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:00:55.202 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-05-06T14:00:55.202 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T14:00:55.202 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:00:55.203 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-05-06T14:00:55.203 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-05-06T14:00:55.203 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T14:00:55.203 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:00:55.203 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw": { 2024-05-06T14:00:55.203 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T14:00:55.203 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:00:55.203 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-05-06T14:00:55.203 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-05-06T14:00:55.203 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 7 2024-05-06T14:00:55.203 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T14:00:55.203 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:00:55.555 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:00:55.555 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T14:00:55.556 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-05-06T14:00:55.556 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading daemons of type(s) crash,osd. Upgrade limited to 1 daemons (1 remaining).", 2024-05-06T14:00:55.556 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [], 2024-05-06T14:00:55.556 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "2/8 daemons upgraded", 2024-05-06T14:00:55.556 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "Currently upgrading osd daemons", 2024-05-06T14:00:55.556 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-05-06T14:00:55.556 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:00:56.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[89212]: from='client.64300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:56.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[89212]: from='client.54372 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:56.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:56.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[89212]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (squid) 2024-05-06T14:00:56.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[89212]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T14:00:56.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:00:56.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:56.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all mgr 2024-05-06T14:00:56.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:56.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:56.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all mon 2024-05-06T14:00:56.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:56.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:56.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all crash 2024-05-06T14:00:56.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:56.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-06T14:00:56.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[89212]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-06T14:00:56.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[89212]: Upgrade: osd.2 is safe to restart 2024-05-06T14:00:56.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[89212]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:56.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/34867057' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:56.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[84247]: from='client.64300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:56.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[84247]: from='client.54372 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:56.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:56.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[84247]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (squid) 2024-05-06T14:00:56.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[84247]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T14:00:56.304 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:00:56.305 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:56.305 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all mgr 2024-05-06T14:00:56.305 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:56.305 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:56.305 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all mon 2024-05-06T14:00:56.305 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:56.305 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:56.305 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all crash 2024-05-06T14:00:56.305 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:56.305 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-06T14:00:56.305 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[84247]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-06T14:00:56.305 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[84247]: Upgrade: osd.2 is safe to restart 2024-05-06T14:00:56.305 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[84247]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:56.306 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:56 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/34867057' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:56.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:56 smithi170 ceph-mon[65866]: from='client.64300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:56.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:56 smithi170 ceph-mon[65866]: from='client.54372 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:56.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:56 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:56.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:56 smithi170 ceph-mon[65866]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (squid) 2024-05-06T14:00:56.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:56 smithi170 ceph-mon[65866]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T14:00:56.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:56 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:00:56.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:56 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:56.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:56 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all mgr 2024-05-06T14:00:56.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:56 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:56.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:56 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:56.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:56 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all mon 2024-05-06T14:00:56.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:56 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:56.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:56 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:56.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:56 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all crash 2024-05-06T14:00:56.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:56 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:56.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:56 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-06T14:00:56.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:56 smithi170 ceph-mon[65866]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-05-06T14:00:56.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:56 smithi170 ceph-mon[65866]: Upgrade: osd.2 is safe to restart 2024-05-06T14:00:56.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:56 smithi170 ceph-mon[65866]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:00:56.328 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:56 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/34867057' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:00:57.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:57 smithi077 ceph-mon[89212]: from='client.64318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:57.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:57 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:00:57.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:57 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:57.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:57 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-06T14:00:57.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:57 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:00:57.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:57 smithi077 ceph-mon[84247]: from='client.64318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:57.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:57 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:00:57.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:57 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:57.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:57 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-06T14:00:57.303 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:57 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:00:57.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:57 smithi170 ceph-mon[65866]: from='client.64318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:00:57.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:57 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:00:57.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:57 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:00:57.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:57 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-06T14:00:57.327 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:57 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:00:58.297 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:58 smithi077 ceph-mon[89212]: Upgrade: Updating osd.2 2024-05-06T14:00:58.297 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:58 smithi077 ceph-mon[89212]: Deploying daemon osd.2 on smithi077 2024-05-06T14:00:58.297 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:00:58 smithi077 ceph-mon[89212]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:00:58.297 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:58 smithi077 ceph-mon[84247]: Upgrade: Updating osd.2 2024-05-06T14:00:58.297 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:58 smithi077 ceph-mon[84247]: Deploying daemon osd.2 on smithi077 2024-05-06T14:00:58.297 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:00:58 smithi077 ceph-mon[84247]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:00:58.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:58 smithi170 ceph-mon[65866]: Upgrade: Updating osd.2 2024-05-06T14:00:58.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:58 smithi170 ceph-mon[65866]: Deploying daemon osd.2 on smithi077 2024-05-06T14:00:58.326 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:00:58 smithi170 ceph-mon[65866]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:01:00.252 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:00 smithi077 ceph-mon[84247]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:00.253 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:00 smithi077 ceph-mon[89212]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:00.253 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:00:59 smithi077 systemd[1]: Stopping Ceph osd.2 for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:01:00.552 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:00 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2[35912]: 2024-05-06T14:01:00.250+0000 7f016364a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:01:00.552 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:00 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2[35912]: 2024-05-06T14:01:00.250+0000 7f016364a700 -1 osd.2 123 *** Got signal Terminated *** 2024-05-06T14:01:00.552 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:00 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2[35912]: 2024-05-06T14:01:00.250+0000 7f016364a700 -1 osd.2 123 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T14:01:00.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:00 smithi170 ceph-mon[65866]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:01.441 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:01 smithi077 ceph-mon[84247]: osd.2 marked itself down and dead 2024-05-06T14:01:01.441 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:01 smithi077 ceph-mon[89212]: osd.2 marked itself down and dead 2024-05-06T14:01:01.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:01 smithi170 ceph-mon[65866]: osd.2 marked itself down and dead 2024-05-06T14:01:02.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:01 smithi077 podman[107556]: 2024-05-06 14:01:01.742807803 +0000 UTC m=+1.612887179 container died 37a61d2951118dd67110025dabcf1f6adc7ac8d0c9416e1b953812a7a7cc55cd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, io.openshift.expose-services=, vendor=Red Hat, Inc., architecture=x86_64, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, name=centos-stream, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, version=8, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, ceph=True) 2024-05-06T14:01:02.394 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:02 smithi077 ceph-mon[84247]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:02.394 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:02 smithi077 ceph-mon[84247]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T14:01:02.395 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:02 smithi077 ceph-mon[84247]: osdmap e124: 8 total, 7 up, 8 in 2024-05-06T14:01:02.395 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:02 smithi077 ceph-mon[89212]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:02.395 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:02 smithi077 ceph-mon[89212]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T14:01:02.395 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:02 smithi077 ceph-mon[89212]: osdmap e124: 8 total, 7 up, 8 in 2024-05-06T14:01:02.396 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:02 smithi077 podman[107556]: 2024-05-06 14:01:02.14321076 +0000 UTC m=+2.013290135 container cleanup 37a61d2951118dd67110025dabcf1f6adc7ac8d0c9416e1b953812a7a7cc55cd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2, vcs-type=git, ceph=True, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, vendor=Red Hat, Inc., GIT_CLEAN=True, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-05-06T14:01:02.396 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:02 smithi077 bash[107556]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2 2024-05-06T14:01:02.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:02 smithi170 ceph-mon[65866]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 132 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:02.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:02 smithi170 ceph-mon[65866]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T14:01:02.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:02 smithi170 ceph-mon[65866]: osdmap e124: 8 total, 7 up, 8 in 2024-05-06T14:01:02.802 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:02 smithi077 podman[107591]: 2024-05-06 14:01:02.394391312 +0000 UTC m=+0.644547102 container remove 37a61d2951118dd67110025dabcf1f6adc7ac8d0c9416e1b953812a7a7cc55cd (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, version=8, io.openshift.expose-services=, com.redhat.component=centos-stream-container, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , ceph=True, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD) 2024-05-06T14:01:03.114 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:02 smithi077 podman[107683]: 2024-05-06 14:01:02.734738458 +0000 UTC m=+0.018501541 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:01:03.114 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:02 smithi077 podman[107683]: 2024-05-06 14:01:02.845538173 +0000 UTC m=+0.129301240 container create 9c10c5ac47036fa69f522437132ed7b09102b6e9d29bd42eaed6ae6d0940c32b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, RELEASE=squid-70ce953, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-05-06T14:01:03.114 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:03 smithi077 podman[107683]: 2024-05-06 14:01:03.031189719 +0000 UTC m=+0.314952790 container init 9c10c5ac47036fa69f522437132ed7b09102b6e9d29bd42eaed6ae6d0940c32b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-70ce953) 2024-05-06T14:01:03.115 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:03 smithi077 podman[107683]: 2024-05-06 14:01:03.034543851 +0000 UTC m=+0.318306929 container start 9c10c5ac47036fa69f522437132ed7b09102b6e9d29bd42eaed6ae6d0940c32b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T14:01:03.115 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:03 smithi077 podman[107683]: 2024-05-06 14:01:03.114562564 +0000 UTC m=+0.398325635 container attach 9c10c5ac47036fa69f522437132ed7b09102b6e9d29bd42eaed6ae6d0940c32b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T14:01:03.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:03 smithi077 ceph-mon[84247]: osdmap e125: 8 total, 7 up, 8 in 2024-05-06T14:01:03.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:03 smithi077 ceph-mon[89212]: osdmap e125: 8 total, 7 up, 8 in 2024-05-06T14:01:03.553 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:03 smithi077 podman[107683]: 2024-05-06 14:01:03.330740359 +0000 UTC m=+0.614503419 container died 9c10c5ac47036fa69f522437132ed7b09102b6e9d29bd42eaed6ae6d0940c32b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430) 2024-05-06T14:01:03.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:03 smithi170 ceph-mon[65866]: osdmap e125: 8 total, 7 up, 8 in 2024-05-06T14:01:03.918 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:01:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:01:03] "GET /metrics HTTP/1.1" 200 38172 "" "Prometheus/2.43.0" 2024-05-06T14:01:04.188 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:03 smithi077 podman[107709]: 2024-05-06 14:01:03.918259943 +0000 UTC m=+0.582541481 container remove 9c10c5ac47036fa69f522437132ed7b09102b6e9d29bd42eaed6ae6d0940c32b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T14:01:04.189 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:03 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.2.service: Deactivated successfully. 2024-05-06T14:01:04.189 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:03 smithi077 systemd[1]: Stopped Ceph osd.2 for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T14:01:04.189 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:03 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.2.service: Consumed 12.479s CPU time. 2024-05-06T14:01:04.189 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:04 smithi077 systemd[1]: Starting Ceph osd.2 for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:01:04.516 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:04 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:01:04.516 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:04 smithi077 ceph-mon[84247]: pgmap v116: 161 pgs: 33 peering, 4 stale+active+clean, 124 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:01:04.516 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:04 smithi077 ceph-mon[84247]: Health check failed: Reduced data availability: 8 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-05-06T14:01:04.517 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:04 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:01:04.517 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:04 smithi077 ceph-mon[89212]: pgmap v116: 161 pgs: 33 peering, 4 stale+active+clean, 124 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:01:04.517 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:04 smithi077 ceph-mon[89212]: Health check failed: Reduced data availability: 8 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-05-06T14:01:04.517 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:04 smithi077 podman[107823]: 2024-05-06 14:01:04.271178707 +0000 UTC m=+0.018643671 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:01:04.517 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:04 smithi077 podman[107823]: 2024-05-06 14:01:04.385990788 +0000 UTC m=+0.133455751 container create 5a3f572ffee0fa67b7d934e296331f30132bb636d8ba36e6b580b3168a71bcdf (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux ) 2024-05-06T14:01:04.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:04 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:01:04.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:04 smithi170 ceph-mon[65866]: pgmap v116: 161 pgs: 33 peering, 4 stale+active+clean, 124 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:01:04.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:04 smithi170 ceph-mon[65866]: Health check failed: Reduced data availability: 8 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-05-06T14:01:04.802 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:04 smithi077 podman[107823]: 2024-05-06 14:01:04.622637428 +0000 UTC m=+0.370102401 container init 5a3f572ffee0fa67b7d934e296331f30132bb636d8ba36e6b580b3168a71bcdf (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2-activate, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-70ce953) 2024-05-06T14:01:04.802 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:04 smithi077 podman[107823]: 2024-05-06 14:01:04.626068571 +0000 UTC m=+0.373533542 container start 5a3f572ffee0fa67b7d934e296331f30132bb636d8ba36e6b580b3168a71bcdf (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, RELEASE=squid-70ce953, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-05-06T14:01:04.803 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:04 smithi077 podman[107823]: 2024-05-06 14:01:04.680367158 +0000 UTC m=+0.427832135 container attach 5a3f572ffee0fa67b7d934e296331f30132bb636d8ba36e6b580b3168a71bcdf (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2-activate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, ceph=True, RELEASE=squid-70ce953, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T14:01:05.383 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:05 smithi077 ceph-mon[84247]: pgmap v117: 161 pgs: 4 active+undersized+degraded, 33 peering, 10 active+undersized, 114 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 25/627 objects degraded (3.987%) 2024-05-06T14:01:05.384 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:05 smithi077 ceph-mon[84247]: Health check failed: Degraded data redundancy: 25/627 objects degraded (3.987%), 4 pgs degraded (PG_DEGRADED) 2024-05-06T14:01:05.384 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:05 smithi077 ceph-mon[89212]: pgmap v117: 161 pgs: 4 active+undersized+degraded, 33 peering, 10 active+undersized, 114 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 25/627 objects degraded (3.987%) 2024-05-06T14:01:05.384 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:05 smithi077 ceph-mon[89212]: Health check failed: Degraded data redundancy: 25/627 objects degraded (3.987%), 4 pgs degraded (PG_DEGRADED) 2024-05-06T14:01:05.384 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2-activate[107839]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-06T14:01:05.384 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2-activate[107839]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-05-06T14:01:05.384 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:05 smithi077 bash[107823]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-06T14:01:05.385 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:05 smithi077 bash[107823]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-05-06T14:01:05.385 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:05 smithi077 bash[107823]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-05-06T14:01:05.385 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:05 smithi077 bash[107823]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-05-06T14:01:05.385 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:05 smithi077 bash[107823]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-05-06T14:01:05.385 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:05 smithi077 bash[107823]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-06T14:01:05.385 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:05 smithi077 bash[107823]: --> ceph-volume raw activate successful for osd ID: 2 2024-05-06T14:01:05.385 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2-activate[107839]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-05-06T14:01:05.385 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2-activate[107839]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-05-06T14:01:05.385 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2-activate[107839]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-05-06T14:01:05.385 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2-activate[107839]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-06T14:01:05.385 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:05 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2-activate[107839]: --> ceph-volume raw activate successful for osd ID: 2 2024-05-06T14:01:05.385 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:05 smithi077 podman[107823]: 2024-05-06 14:01:05.09373226 +0000 UTC m=+0.841197218 container died 5a3f572ffee0fa67b7d934e296331f30132bb636d8ba36e6b580b3168a71bcdf (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2-activate, RELEASE=squid-70ce953, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-05-06T14:01:05.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:05 smithi170 ceph-mon[65866]: pgmap v117: 161 pgs: 4 active+undersized+degraded, 33 peering, 10 active+undersized, 114 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 25/627 objects degraded (3.987%) 2024-05-06T14:01:05.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:05 smithi170 ceph-mon[65866]: Health check failed: Degraded data redundancy: 25/627 objects degraded (3.987%), 4 pgs degraded (PG_DEGRADED) 2024-05-06T14:01:06.036 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:05 smithi077 podman[107970]: 2024-05-06 14:01:05.684362805 +0000 UTC m=+0.585441531 container remove 5a3f572ffee0fa67b7d934e296331f30132bb636d8ba36e6b580b3168a71bcdf (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2-activate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-70ce953) 2024-05-06T14:01:06.036 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:05 smithi077 podman[108034]: 2024-05-06 14:01:05.800496607 +0000 UTC m=+0.018175529 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:01:06.036 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:05 smithi077 podman[108034]: 2024-05-06 14:01:05.918583449 +0000 UTC m=+0.136262369 container create a4c3266b87f88c36d6b9f7be446856c82df87e390d25e49f7b58f32b163d12d3 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS) 2024-05-06T14:01:06.552 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:06 smithi077 podman[108034]: 2024-05-06 14:01:06.120846196 +0000 UTC m=+0.338525118 container init a4c3266b87f88c36d6b9f7be446856c82df87e390d25e49f7b58f32b163d12d3 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-70ce953, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T14:01:06.552 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:06 smithi077 podman[108034]: 2024-05-06 14:01:06.123823543 +0000 UTC m=+0.341502465 container start a4c3266b87f88c36d6b9f7be446856c82df87e390d25e49f7b58f32b163d12d3 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, GIT_CLEAN=True, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-06T14:01:06.552 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:06 smithi077 ceph-osd[108052]: -- 172.21.15.77:0/1234222816 <== mon.0 v2:172.21.15.77:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55da95219680 con 0x55da951e2c00 2024-05-06T14:01:06.552 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:06 smithi077 bash[108034]: a4c3266b87f88c36d6b9f7be446856c82df87e390d25e49f7b58f32b163d12d3 2024-05-06T14:01:06.553 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:06 smithi077 systemd[1]: Started Ceph osd.2 for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T14:01:07.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:06 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2[108048]: 2024-05-06T14:01:06.660+0000 7fb20a2f87c0 -1 Falling back to public interface 2024-05-06T14:01:07.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:07 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:07.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:07 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:07.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:07 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:07.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:07 smithi077 ceph-mon[84247]: pgmap v118: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 56/627 objects degraded (8.931%) 2024-05-06T14:01:07.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:07 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:07.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:07 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:07.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:07 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:07.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:07 smithi077 ceph-mon[89212]: pgmap v118: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 56/627 objects degraded (8.931%) 2024-05-06T14:01:07.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:07 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:07.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:07 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:07.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:07 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:07.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:07 smithi170 ceph-mon[65866]: pgmap v118: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 56/627 objects degraded (8.931%) 2024-05-06T14:01:08.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:08 smithi077 ceph-mon[89212]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 8 pgs peering) 2024-05-06T14:01:08.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:08 smithi077 ceph-mon[84247]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 8 pgs peering) 2024-05-06T14:01:08.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:08 smithi170 ceph-mon[65866]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 8 pgs peering) 2024-05-06T14:01:09.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:09 smithi077 ceph-mon[84247]: pgmap v119: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 56/627 objects degraded (8.931%) 2024-05-06T14:01:09.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:09 smithi077 ceph-mon[89212]: pgmap v119: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 56/627 objects degraded (8.931%) 2024-05-06T14:01:09.552 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:09 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2[108048]: 2024-05-06T14:01:09.365+0000 7fb20a2f87c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-05-06T14:01:09.553 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:09 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2[108048]: 2024-05-06T14:01:09.392+0000 7fb20a2f87c0 -1 osd.2 123 log_to_monitors true 2024-05-06T14:01:09.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:09 smithi170 ceph-mon[65866]: pgmap v119: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 133 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 56/627 objects degraded (8.931%) 2024-05-06T14:01:10.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:10 smithi077 ceph-mon[84247]: from='osd.2 [v2:172.21.15.77:6818/4269386453,v1:172.21.15.77:6819/4269386453]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-06T14:01:10.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:10 smithi077 ceph-mon[84247]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-06T14:01:10.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:10 smithi077 ceph-mon[89212]: from='osd.2 [v2:172.21.15.77:6818/4269386453,v1:172.21.15.77:6819/4269386453]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-06T14:01:10.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:10 smithi077 ceph-mon[89212]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-06T14:01:10.553 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:01:10 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2[108048]: 2024-05-06T14:01:10.307+0000 7fb1fb74f700 -1 osd.2 123 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T14:01:10.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:10 smithi170 ceph-mon[65866]: from='osd.2 [v2:172.21.15.77:6818/4269386453,v1:172.21.15.77:6819/4269386453]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-06T14:01:10.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:10 smithi170 ceph-mon[65866]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-06T14:01:11.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:11 smithi077 ceph-mon[84247]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-06T14:01:11.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:11 smithi077 ceph-mon[84247]: osdmap e126: 8 total, 7 up, 8 in 2024-05-06T14:01:11.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:11 smithi077 ceph-mon[84247]: from='osd.2 [v2:172.21.15.77:6818/4269386453,v1:172.21.15.77:6819/4269386453]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T14:01:11.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:11 smithi077 ceph-mon[84247]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T14:01:11.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:11 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:11.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:11 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:11.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:11 smithi077 ceph-mon[84247]: pgmap v121: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 56/627 objects degraded (8.931%) 2024-05-06T14:01:11.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:11 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:11.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:11 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:01:11.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:11 smithi077 ceph-mon[89212]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-06T14:01:11.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:11 smithi077 ceph-mon[89212]: osdmap e126: 8 total, 7 up, 8 in 2024-05-06T14:01:11.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:11 smithi077 ceph-mon[89212]: from='osd.2 [v2:172.21.15.77:6818/4269386453,v1:172.21.15.77:6819/4269386453]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T14:01:11.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:11 smithi077 ceph-mon[89212]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T14:01:11.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:11 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:11.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:11 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:11.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:11 smithi077 ceph-mon[89212]: pgmap v121: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 56/627 objects degraded (8.931%) 2024-05-06T14:01:11.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:11 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:11.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:11 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:01:11.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:11 smithi170 ceph-mon[65866]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-06T14:01:11.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:11 smithi170 ceph-mon[65866]: osdmap e126: 8 total, 7 up, 8 in 2024-05-06T14:01:11.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:11 smithi170 ceph-mon[65866]: from='osd.2 [v2:172.21.15.77:6818/4269386453,v1:172.21.15.77:6819/4269386453]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T14:01:11.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:11 smithi170 ceph-mon[65866]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T14:01:11.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:11 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:11.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:11 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:11.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:11 smithi170 ceph-mon[65866]: pgmap v121: 161 pgs: 14 active+undersized+degraded, 33 active+undersized, 114 active+clean; 585 KiB data, 148 MiB used, 715 GiB / 715 GiB avail; 56/627 objects degraded (8.931%) 2024-05-06T14:01:11.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:11 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:11.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:11 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:01:12.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:12 smithi077 ceph-mon[84247]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T14:01:12.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:12 smithi077 ceph-mon[84247]: osd.2 [v2:172.21.15.77:6818/4269386453,v1:172.21.15.77:6819/4269386453] boot 2024-05-06T14:01:12.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:12 smithi077 ceph-mon[84247]: osdmap e127: 8 total, 8 up, 8 in 2024-05-06T14:01:12.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:12 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T14:01:12.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:12 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:12.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:12 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:12.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:12 smithi077 ceph-mon[84247]: osdmap e128: 8 total, 8 up, 8 in 2024-05-06T14:01:12.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:12 smithi077 ceph-mon[89212]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T14:01:12.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:12 smithi077 ceph-mon[89212]: osd.2 [v2:172.21.15.77:6818/4269386453,v1:172.21.15.77:6819/4269386453] boot 2024-05-06T14:01:12.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:12 smithi077 ceph-mon[89212]: osdmap e127: 8 total, 8 up, 8 in 2024-05-06T14:01:12.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:12 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T14:01:12.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:12 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:12.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:12 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:12.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:12 smithi077 ceph-mon[89212]: osdmap e128: 8 total, 8 up, 8 in 2024-05-06T14:01:12.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:12 smithi170 ceph-mon[65866]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T14:01:12.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:12 smithi170 ceph-mon[65866]: osd.2 [v2:172.21.15.77:6818/4269386453,v1:172.21.15.77:6819/4269386453] boot 2024-05-06T14:01:12.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:12 smithi170 ceph-mon[65866]: osdmap e127: 8 total, 8 up, 8 in 2024-05-06T14:01:12.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:12 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-06T14:01:12.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:12 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:12.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:12 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:12.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:12 smithi170 ceph-mon[65866]: osdmap e128: 8 total, 8 up, 8 in 2024-05-06T14:01:13.566 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:13 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:01:13.566 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:13 smithi077 ceph-mon[84247]: Health check update: Degraded data redundancy: 56/627 objects degraded (8.931%), 14 pgs degraded (PG_DEGRADED) 2024-05-06T14:01:13.566 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:13 smithi077 ceph-mon[84247]: pgmap v124: 161 pgs: 25 peering, 4 active+undersized+degraded, 18 active+undersized, 114 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 25/627 objects degraded (3.987%) 2024-05-06T14:01:13.567 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:13 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:01:13.567 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:13 smithi077 ceph-mon[89212]: Health check update: Degraded data redundancy: 56/627 objects degraded (8.931%), 14 pgs degraded (PG_DEGRADED) 2024-05-06T14:01:13.567 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:13 smithi077 ceph-mon[89212]: pgmap v124: 161 pgs: 25 peering, 4 active+undersized+degraded, 18 active+undersized, 114 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 25/627 objects degraded (3.987%) 2024-05-06T14:01:13.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:13 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:01:13.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:13 smithi170 ceph-mon[65866]: Health check update: Degraded data redundancy: 56/627 objects degraded (8.931%), 14 pgs degraded (PG_DEGRADED) 2024-05-06T14:01:13.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:13 smithi170 ceph-mon[65866]: pgmap v124: 161 pgs: 25 peering, 4 active+undersized+degraded, 18 active+undersized, 114 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 25/627 objects degraded (3.987%) 2024-05-06T14:01:13.833 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:01:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:01:13] "GET /metrics HTTP/1.1" 200 38033 "" "Prometheus/2.43.0" 2024-05-06T14:01:16.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: pgmap v125: 161 pgs: 25 peering, 1 active+undersized+degraded, 11 active+undersized, 124 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 1/627 objects degraded (0.159%) 2024-05-06T14:01:16.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:16.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:16.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:01:16.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:01:16.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:16.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:01:16.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:16.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:16.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:16.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:16.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:16.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:16.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:16.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:16.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:16.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-05-06T14:01:16.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-05-06T14:01:16.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:01:16.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-05-06T14:01:16.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-05-06T14:01:16.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-05-06T14:01:16.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-05-06T14:01:16.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-05-06T14:01:16.554 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-05-06T14:01:16.554 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: pgmap v125: 161 pgs: 25 peering, 1 active+undersized+degraded, 11 active+undersized, 124 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 1/627 objects degraded (0.159%) 2024-05-06T14:01:16.555 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:16.555 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:16.555 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:01:16.555 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:01:16.555 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:16.555 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:01:16.555 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:16.555 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:16.555 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:16.555 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:16.555 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:16.555 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:16.555 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:16.555 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:16.555 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:16.556 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-05-06T14:01:16.556 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-05-06T14:01:16.556 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:01:16.556 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-05-06T14:01:16.556 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-05-06T14:01:16.556 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-05-06T14:01:16.556 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-05-06T14:01:16.556 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-05-06T14:01:16.556 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-05-06T14:01:16.556 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-05-06T14:01:16.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:01:16.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-05-06T14:01:16.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-05-06T14:01:16.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-05-06T14:01:16.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-05-06T14:01:16.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-05-06T14:01:16.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:01:16.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:01:16.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:01:16.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:01:16.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:01:16.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:01:16.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-05-06T14:01:16.557 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-05-06T14:01:16.558 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:01:16.558 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:01:16.558 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:16.558 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:01:16.558 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:01:16.558 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:16.558 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-05-06T14:01:16.558 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:01:16.558 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-05-06T14:01:16.558 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-05-06T14:01:16.558 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-05-06T14:01:16.558 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-05-06T14:01:16.559 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-05-06T14:01:16.559 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:01:16.559 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:01:16.559 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:01:16.559 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:01:16.559 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:01:16.559 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:01:16.559 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-05-06T14:01:16.559 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-05-06T14:01:16.559 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:01:16.559 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:01:16.559 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:16.559 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:01:16.559 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:01:16.559 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:16 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:16.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: pgmap v125: 161 pgs: 25 peering, 1 active+undersized+degraded, 11 active+undersized, 124 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 1/627 objects degraded (0.159%) 2024-05-06T14:01:16.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:16.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:16.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:01:16.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:01:16.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:16.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:01:16.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:16.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:16.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:16.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:16.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:16.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:16.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:16.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:16.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:16.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-05-06T14:01:16.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-05-06T14:01:16.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:01:16.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-05-06T14:01:16.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-05-06T14:01:16.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-05-06T14:01:16.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-05-06T14:01:16.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-05-06T14:01:16.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-05-06T14:01:16.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-05-06T14:01:16.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:01:16.578 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-05-06T14:01:16.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-05-06T14:01:16.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-05-06T14:01:16.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-05-06T14:01:16.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-05-06T14:01:16.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:01:16.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:01:16.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:01:16.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:01:16.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:01:16.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:01:16.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-05-06T14:01:16.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-05-06T14:01:16.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:01:16.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:01:16.579 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:16.580 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:01:16.580 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:01:16.580 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:16 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:17.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:17 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all nvmeof 2024-05-06T14:01:17.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:17 smithi077 ceph-mon[84247]: Upgrade: Finalizing container_image settings 2024-05-06T14:01:17.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:17 smithi077 ceph-mon[84247]: Upgrade: Complete! 2024-05-06T14:01:17.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:17 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:17.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:17 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all nvmeof 2024-05-06T14:01:17.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:17 smithi077 ceph-mon[89212]: Upgrade: Finalizing container_image settings 2024-05-06T14:01:17.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:17 smithi077 ceph-mon[89212]: Upgrade: Complete! 2024-05-06T14:01:17.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:17 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:17.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:17 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all nvmeof 2024-05-06T14:01:17.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:17 smithi170 ceph-mon[65866]: Upgrade: Finalizing container_image settings 2024-05-06T14:01:17.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:17 smithi170 ceph-mon[65866]: Upgrade: Complete! 2024-05-06T14:01:17.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:17 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:18.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:18 smithi077 ceph-mon[89212]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-05-06T14:01:18.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:18 smithi077 ceph-mon[89212]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/627 objects degraded (0.159%), 1 pg degraded) 2024-05-06T14:01:18.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:18 smithi077 ceph-mon[89212]: Cluster is now healthy 2024-05-06T14:01:18.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:18 smithi077 ceph-mon[84247]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-05-06T14:01:18.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:18 smithi077 ceph-mon[84247]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/627 objects degraded (0.159%), 1 pg degraded) 2024-05-06T14:01:18.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:18 smithi077 ceph-mon[84247]: Cluster is now healthy 2024-05-06T14:01:18.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:18 smithi170 ceph-mon[65866]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-05-06T14:01:18.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:18 smithi170 ceph-mon[65866]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/627 objects degraded (0.159%), 1 pg degraded) 2024-05-06T14:01:18.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:18 smithi170 ceph-mon[65866]: Cluster is now healthy 2024-05-06T14:01:20.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:20 smithi077 ceph-mon[84247]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:01:20.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:20 smithi077 ceph-mon[89212]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:01:20.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:20 smithi170 ceph-mon[65866]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:01:22.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:22 smithi077 ceph-mon[89212]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T14:01:22.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:22 smithi077 ceph-mon[84247]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T14:01:22.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:22 smithi170 ceph-mon[65866]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T14:01:24.052 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:01:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:01:23] "GET /metrics HTTP/1.1" 200 38033 "" "Prometheus/2.43.0" 2024-05-06T14:01:24.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:24 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:01:24.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:24 smithi077 ceph-mon[84247]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 944 B/s rd, 0 op/s 2024-05-06T14:01:24.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:24 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:01:24.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:24 smithi077 ceph-mon[89212]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 944 B/s rd, 0 op/s 2024-05-06T14:01:24.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:24 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:01:24.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:24 smithi170 ceph-mon[65866]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 944 B/s rd, 0 op/s 2024-05-06T14:01:26.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:26 smithi077 ceph-mon[84247]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:26.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:26 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:26.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:26 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:01:26.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:26 smithi077 ceph-mon[89212]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:26.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:26 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:26.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:26 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:01:26.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:26 smithi170 ceph-mon[65866]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:26.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:26 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:26.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:26 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:01:26.784 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch ps' 2024-05-06T14:01:27.412 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:27 smithi077 ceph-mon[84247]: from='client.64330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:01:27.413 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:27 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:27.413 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:27 smithi077 ceph-mon[89212]: from='client.64330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:01:27.413 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:27 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:27.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:27 smithi170 ceph-mon[65866]: from='client.64330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:01:27.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:27 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:28.409 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:28 smithi077 ceph-mon[84247]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:01:28.410 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:28 smithi077 ceph-mon[89212]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:01:28.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:28 smithi170 ceph-mon[65866]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:01:28.854 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T14:01:28.854 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (6m) 18s ago 28m 26.6M - 0.25.0 c8568f914cd2 997175e3e64e 2024-05-06T14:01:28.854 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (5m) 3m ago 28m 80.6M - 9.4.12 2bacad6d85d8 140bb9a2760a 2024-05-06T14:01:28.854 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (6m) 18s ago 27m 64.1M - 3.5 e1d6a67b021e 9353a303fb55 2024-05-06T14:01:28.855 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283,8765 running (7m) 3m ago 33m 447M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 125521be53a3 2024-05-06T14:01:28.855 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (17m) 18s ago 34m 507M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T14:01:28.855 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (3m) 18s ago 34m 48.0M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 5e50731ccbe8 2024-05-06T14:01:28.855 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (4m) 3m ago 33m 36.3M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 61cd48b3d0ff 2024-05-06T14:01:28.855 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (3m) 18s ago 33m 39.5M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 8a473cc2ffd1 2024-05-06T14:01:28.855 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (7m) 18s ago 29m 15.8M - 1.5.0 0da6a335fe13 67f45fa4968b 2024-05-06T14:01:28.855 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (7m) 3m ago 29m 16.4M - 1.5.0 0da6a335fe13 02081b71a5dd 2024-05-06T14:01:28.855 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (109s) 18s ago 32m 43.3M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 fbe876a933ed 2024-05-06T14:01:28.855 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (74s) 18s ago 32m 43.8M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 6f664e567c93 2024-05-06T14:01:28.855 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (23s) 18s ago 32m 15.1M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 a4c3266b87f8 2024-05-06T14:01:28.855 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (31m) 18s ago 31m 55.6M 2992M 17.2.0 e1d6a67b021e 8feb7c077aaa 2024-05-06T14:01:28.855 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (31m) 3m ago 31m 58.5M 3248M 17.2.0 e1d6a67b021e f865acc44a06 2024-05-06T14:01:28.855 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (30m) 3m ago 30m 52.8M 3248M 17.2.0 e1d6a67b021e a55bd4122f6f 2024-05-06T14:01:28.855 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (30m) 3m ago 30m 51.4M 3248M 17.2.0 e1d6a67b021e 362938a0b357 2024-05-06T14:01:28.856 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (30m) 3m ago 30m 54.1M 3248M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T14:01:28.856 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (6m) 3m ago 29m 60.1M - 2.43.0 a07b618ecd1d 10d5c513bb9b 2024-05-06T14:01:28.856 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (27m) 18s ago 27m 79.2M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T14:01:28.856 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (27m) 3m ago 27m 77.2M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T14:01:29.661 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 2'"'"'' 2024-05-06T14:01:30.217 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:30 smithi077 ceph-mon[84247]: from='client.64336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:01:30.218 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:30 smithi077 ceph-mon[84247]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:30.218 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:30 smithi077 ceph-mon[89212]: from='client.64336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:01:30.218 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:30 smithi077 ceph-mon[89212]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:30.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:30 smithi170 ceph-mon[65866]: from='client.64336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:01:30.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:30 smithi170 ceph-mon[65866]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:31.819 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T14:01:32.448 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:32 smithi077 ceph-mon[84247]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:32.449 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:32 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/520820057' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:32.449 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:32 smithi077 ceph-mon[89212]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:32.449 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:32 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/520820057' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:32.475 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 8'"'"'' 2024-05-06T14:01:32.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:32 smithi170 ceph-mon[65866]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:32.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:32 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/520820057' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:33.991 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:01:33 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:01:33] "GET /metrics HTTP/1.1" 200 38180 "" "Prometheus/2.43.0" 2024-05-06T14:01:34.293 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:34 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:01:34.293 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:34 smithi077 ceph-mon[84247]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:01:34.294 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:34 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:01:34.294 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:34 smithi077 ceph-mon[89212]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:01:34.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:34 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:01:34.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:34 smithi170 ceph-mon[65866]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:01:36.118 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T14:01:36.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:36 smithi077 ceph-mon[89212]: from='client.64348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:01:36.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:36 smithi077 ceph-mon[89212]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:36.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:36 smithi077 ceph-mon[84247]: from='client.64348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:01:36.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:36 smithi077 ceph-mon[84247]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:36.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:36 smithi170 ceph-mon[65866]: from='client.64348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:01:36.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:36 smithi170 ceph-mon[65866]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:36.880 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch upgrade status' 2024-05-06T14:01:38.417 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:38 smithi077 ceph-mon[84247]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:01:38.418 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:38 smithi077 ceph-mon[89212]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:01:38.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:38 smithi170 ceph-mon[65866]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:01:38.845 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:01:38.845 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": null, 2024-05-06T14:01:38.845 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": false, 2024-05-06T14:01:38.845 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "", 2024-05-06T14:01:38.846 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [], 2024-05-06T14:01:38.846 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": null, 2024-05-06T14:01:38.846 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "", 2024-05-06T14:01:38.846 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-05-06T14:01:38.846 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:01:39.473 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:39 smithi077 ceph-mon[84247]: from='client.64354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:01:39.473 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:39 smithi077 ceph-mon[84247]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:39.474 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:39 smithi077 ceph-mon[89212]: from='client.64354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:01:39.474 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:39 smithi077 ceph-mon[89212]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:39.475 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph health detail' 2024-05-06T14:01:39.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:39 smithi170 ceph-mon[65866]: from='client.64354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:01:39.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:39 smithi170 ceph-mon[65866]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:41.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:01:41.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:01:41.554 INFO:teuthology.orchestra.run.smithi077.stdout:HEALTH_OK 2024-05-06T14:01:41.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:01:42.300 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:42 smithi077 ceph-mon[84247]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:42.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:42 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/2557382934' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T14:01:42.301 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:42 smithi077 ceph-mon[89212]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:42.301 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:42 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/2557382934' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T14:01:42.325 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd' 2024-05-06T14:01:42.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:42 smithi170 ceph-mon[65866]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:42.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:42 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/2557382934' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T14:01:43.837 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:01:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:01:43] "GET /metrics HTTP/1.1" 200 38182 "" "Prometheus/2.43.0" 2024-05-06T14:01:44.503 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:44 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:01:44.503 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:44 smithi077 ceph-mon[84247]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:01:44.503 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:44 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:01:44.503 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:44 smithi077 ceph-mon[89212]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:01:44.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:44 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:01:44.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:44 smithi170 ceph-mon[65866]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:01:45.933 INFO:teuthology.orchestra.run.smithi077.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:01:46.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:46 smithi077 ceph-mon[89212]: from='client.64366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:01:46.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:46 smithi077 ceph-mon[89212]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:46.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:46 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:46.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:46 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:01:46.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:46 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:01:46.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:46 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:46.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:46 smithi077 ceph-mon[84247]: from='client.64366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:01:46.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:46 smithi077 ceph-mon[84247]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:46.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:46 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:46.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:46 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:01:46.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:46 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:01:46.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:46 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:46.574 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-05-06T14:01:46.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:46 smithi170 ceph-mon[65866]: from='client.64366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:01:46.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:46 smithi170 ceph-mon[65866]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:46.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:46 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:46.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:46 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:01:46.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:46 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:01:46.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:46 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:47.478 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:47 smithi077 ceph-mon[84247]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:01:47.478 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:47 smithi077 ceph-mon[84247]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:01:47.478 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:47 smithi077 ceph-mon[89212]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:01:47.478 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:47 smithi077 ceph-mon[89212]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:01:47.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:47 smithi170 ceph-mon[65866]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:01:47.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:47 smithi170 ceph-mon[65866]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:01:48.452 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:48 smithi077 ceph-mon[84247]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:01:48.453 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:48 smithi077 ceph-mon[89212]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:01:48.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:48 smithi170 ceph-mon[65866]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:01:49.572 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T14:01:49.686 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:49.686 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[89212]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (squid) 2024-05-06T14:01:49.686 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[89212]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T14:01:49.686 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:01:49.686 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:49.686 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all mgr 2024-05-06T14:01:49.686 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:49.687 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:49.687 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all mon 2024-05-06T14:01:49.687 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:49.687 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:49.687 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all crash 2024-05-06T14:01:49.687 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:49.687 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-06T14:01:49.687 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[89212]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-06T14:01:49.687 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[89212]: Upgrade: osd.3 is safe to restart 2024-05-06T14:01:49.687 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[89212]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:49.687 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:49.687 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[84247]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (squid) 2024-05-06T14:01:49.687 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[84247]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T14:01:49.688 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:01:49.688 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:49.688 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all mgr 2024-05-06T14:01:49.688 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:49.688 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:49.688 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all mon 2024-05-06T14:01:49.688 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:49.688 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:49.688 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all crash 2024-05-06T14:01:49.688 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:49.688 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-06T14:01:49.688 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[84247]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-06T14:01:49.688 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[84247]: Upgrade: osd.3 is safe to restart 2024-05-06T14:01:49.688 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:49 smithi077 ceph-mon[84247]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:49.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:49 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:49.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:49 smithi170 ceph-mon[65866]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (squid) 2024-05-06T14:01:49.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:49 smithi170 ceph-mon[65866]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T14:01:49.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:49 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:01:49.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:49 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:49.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:49 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all mgr 2024-05-06T14:01:49.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:49 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:49.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:49 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:49.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:49 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all mon 2024-05-06T14:01:49.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:49 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:49.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:49 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:49.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:49 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all crash 2024-05-06T14:01:49.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:49 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:49.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:49 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-06T14:01:49.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:49 smithi170 ceph-mon[65866]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-05-06T14:01:49.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:49 smithi170 ceph-mon[65866]: Upgrade: osd.3 is safe to restart 2024-05-06T14:01:49.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:49 smithi170 ceph-mon[65866]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:50.236 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T14:01:50.236 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (6m) 39s ago 29m 26.6M - 0.25.0 c8568f914cd2 997175e3e64e 2024-05-06T14:01:50.236 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (6m) 4m ago 28m 80.6M - 9.4.12 2bacad6d85d8 140bb9a2760a 2024-05-06T14:01:50.236 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (7m) 39s ago 28m 64.1M - 3.5 e1d6a67b021e 9353a303fb55 2024-05-06T14:01:50.236 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283,8765 running (7m) 4m ago 33m 447M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 125521be53a3 2024-05-06T14:01:50.236 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (17m) 39s ago 35m 507M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T14:01:50.237 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (4m) 39s ago 35m 48.0M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 5e50731ccbe8 2024-05-06T14:01:50.237 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (5m) 4m ago 33m 36.3M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 61cd48b3d0ff 2024-05-06T14:01:50.237 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (3m) 39s ago 33m 39.5M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 8a473cc2ffd1 2024-05-06T14:01:50.237 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (7m) 39s ago 29m 15.8M - 1.5.0 0da6a335fe13 67f45fa4968b 2024-05-06T14:01:50.237 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (7m) 4m ago 29m 16.4M - 1.5.0 0da6a335fe13 02081b71a5dd 2024-05-06T14:01:50.237 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (2m) 39s ago 33m 43.3M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 fbe876a933ed 2024-05-06T14:01:50.237 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (95s) 39s ago 32m 43.8M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 6f664e567c93 2024-05-06T14:01:50.237 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (44s) 39s ago 32m 15.1M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 a4c3266b87f8 2024-05-06T14:01:50.237 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (32m) 39s ago 32m 55.6M 2992M 17.2.0 e1d6a67b021e 8feb7c077aaa 2024-05-06T14:01:50.237 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (31m) 4m ago 31m 58.5M 3248M 17.2.0 e1d6a67b021e f865acc44a06 2024-05-06T14:01:50.237 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (31m) 4m ago 31m 52.8M 3248M 17.2.0 e1d6a67b021e a55bd4122f6f 2024-05-06T14:01:50.237 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (30m) 4m ago 30m 51.4M 3248M 17.2.0 e1d6a67b021e 362938a0b357 2024-05-06T14:01:50.237 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (30m) 4m ago 30m 54.1M 3248M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T14:01:50.237 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (7m) 4m ago 29m 60.1M - 2.43.0 a07b618ecd1d 10d5c513bb9b 2024-05-06T14:01:50.237 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (28m) 39s ago 28m 79.2M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T14:01:50.237 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (28m) 4m ago 28m 77.2M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T14:01:50.644 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:01:50.644 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-05-06T14:01:50.644 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T14:01:50.645 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:01:50.645 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-05-06T14:01:50.645 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T14:01:50.645 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:01:50.645 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-05-06T14:01:50.645 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-05-06T14:01:50.645 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T14:01:50.645 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:01:50.645 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw": { 2024-05-06T14:01:50.645 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T14:01:50.645 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:01:50.645 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-05-06T14:01:50.645 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-05-06T14:01:50.645 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 8 2024-05-06T14:01:50.645 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T14:01:50.646 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:01:50.801 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:50 smithi077 ceph-mon[84247]: from='client.64372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:01:50.801 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:50 smithi077 ceph-mon[84247]: from='client.64378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:01:50.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:50 smithi077 ceph-mon[89212]: from='client.64372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:01:50.802 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:50 smithi077 ceph-mon[89212]: from='client.64378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:01:50.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:50 smithi170 ceph-mon[65866]: from='client.64372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:01:50.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:50 smithi170 ceph-mon[65866]: from='client.64378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:01:50.993 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:01:50.993 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T14:01:50.993 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-05-06T14:01:50.993 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-05-06T14:01:50.993 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [], 2024-05-06T14:01:50.993 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "3/8 daemons upgraded", 2024-05-06T14:01:50.994 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "Currently upgrading osd daemons", 2024-05-06T14:01:50.994 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-05-06T14:01:50.994 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:01:51.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:51 smithi077 ceph-mon[84247]: from='client.54450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:01:51.495 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:51 smithi077 ceph-mon[84247]: Upgrade: Updating osd.3 2024-05-06T14:01:51.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:51 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:51.802 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:51 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-06T14:01:51.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:51 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:01:51.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:51 smithi077 ceph-mon[84247]: Deploying daemon osd.3 on smithi077 2024-05-06T14:01:51.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:51 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/2592287118' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:51.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:51 smithi077 ceph-mon[84247]: from='client.54462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:01:51.803 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:51 smithi077 ceph-mon[84247]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:51.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:51 smithi077 ceph-mon[89212]: from='client.54450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:01:51.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:51 smithi077 ceph-mon[89212]: Upgrade: Updating osd.3 2024-05-06T14:01:51.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:51 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:51.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:51 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-06T14:01:51.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:51 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:01:51.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:51 smithi077 ceph-mon[89212]: Deploying daemon osd.3 on smithi077 2024-05-06T14:01:51.803 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:51 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/2592287118' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:51.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:51 smithi077 ceph-mon[89212]: from='client.54462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:01:51.804 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:51 smithi077 ceph-mon[89212]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:51.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:51 smithi170 ceph-mon[65866]: from='client.54450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:01:51.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:51 smithi170 ceph-mon[65866]: Upgrade: Updating osd.3 2024-05-06T14:01:51.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:51 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:51.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:51 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-06T14:01:51.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:51 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:01:51.826 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:51 smithi170 ceph-mon[65866]: Deploying daemon osd.3 on smithi077 2024-05-06T14:01:51.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:51 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/2592287118' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:01:51.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:51 smithi170 ceph-mon[65866]: from='client.54462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:01:51.827 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:51 smithi170 ceph-mon[65866]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:01:53.993 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:53 smithi077 systemd[1]: Stopping Ceph osd.3 for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:01:53.993 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:01:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:01:53] "GET /metrics HTTP/1.1" 200 38182 "" "Prometheus/2.43.0" 2024-05-06T14:01:54.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:54 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:01:54.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:54 smithi077 ceph-mon[84247]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:01:54.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:54 smithi077 ceph-mon[84247]: osd.3 marked itself down and dead 2024-05-06T14:01:54.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:54 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:01:54.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:54 smithi077 ceph-mon[89212]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:01:54.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:54 smithi077 ceph-mon[89212]: osd.3 marked itself down and dead 2024-05-06T14:01:54.302 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3[38889]: 2024-05-06T14:01:53.992+0000 7f260c05b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:01:54.302 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3[38889]: 2024-05-06T14:01:53.992+0000 7f260c05b700 -1 osd.3 128 *** Got signal Terminated *** 2024-05-06T14:01:54.302 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3[38889]: 2024-05-06T14:01:53.992+0000 7f260c05b700 -1 osd.3 128 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T14:01:54.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:54 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:01:54.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:54 smithi170 ceph-mon[65866]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:01:54.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:54 smithi170 ceph-mon[65866]: osd.3 marked itself down and dead 2024-05-06T14:01:54.867 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:54 smithi077 podman[114044]: 2024-05-06 14:01:54.605535446 +0000 UTC m=+0.727429339 container died 8feb7c077aaac156cb9b6fcd6a3aa9d4181efcf84b092900e56b2409f94e7885 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, release=754, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, ceph=True, io.k8s.display-name=CentOS Stream 8, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, maintainer=Guillaume Abrioux , RELEASE=HEAD, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-05-06T14:01:55.184 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:54 smithi077 podman[114044]: 2024-05-06 14:01:54.967784788 +0000 UTC m=+1.089678685 container cleanup 8feb7c077aaac156cb9b6fcd6a3aa9d4181efcf84b092900e56b2409f94e7885 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, name=centos-stream, vcs-type=git, RELEASE=HEAD, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-05-06T14:01:55.184 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:54 smithi077 bash[114044]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3 2024-05-06T14:01:55.523 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:55 smithi077 ceph-mon[84247]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T14:01:55.523 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:55 smithi077 ceph-mon[84247]: osdmap e129: 8 total, 7 up, 8 in 2024-05-06T14:01:55.524 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:55 smithi077 ceph-mon[89212]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T14:01:55.524 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:55 smithi077 ceph-mon[89212]: osdmap e129: 8 total, 7 up, 8 in 2024-05-06T14:01:55.524 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:55 smithi077 podman[114061]: 2024-05-06 14:01:55.244115715 +0000 UTC m=+0.639398561 container remove 8feb7c077aaac156cb9b6fcd6a3aa9d4181efcf84b092900e56b2409f94e7885 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3, io.buildah.version=1.19.8, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, ceph=True, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_BRANCH=HEAD) 2024-05-06T14:01:55.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:55 smithi170 ceph-mon[65866]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T14:01:55.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:55 smithi170 ceph-mon[65866]: osdmap e129: 8 total, 7 up, 8 in 2024-05-06T14:01:55.788 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:55 smithi077 podman[114152]: 2024-05-06 14:01:55.484265541 +0000 UTC m=+0.021854643 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:01:55.788 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:55 smithi077 podman[114152]: 2024-05-06 14:01:55.595452888 +0000 UTC m=+0.133041989 container create 2b6dce514a6796a7db6b89b496a8a6db2edf5cc21f0e177e16d099f857e243a1 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True) 2024-05-06T14:01:56.051 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:55 smithi077 podman[114152]: 2024-05-06 14:01:55.78937847 +0000 UTC m=+0.326967573 container init 2b6dce514a6796a7db6b89b496a8a6db2edf5cc21f0e177e16d099f857e243a1 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T14:01:56.052 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:55 smithi077 podman[114152]: 2024-05-06 14:01:55.792628567 +0000 UTC m=+0.330217679 container start 2b6dce514a6796a7db6b89b496a8a6db2edf5cc21f0e177e16d099f857e243a1 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-05-06T14:01:56.052 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:55 smithi077 podman[114152]: 2024-05-06 14:01:55.864342942 +0000 UTC m=+0.401932062 container attach 2b6dce514a6796a7db6b89b496a8a6db2edf5cc21f0e177e16d099f857e243a1 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-06T14:01:56.425 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:56 smithi077 ceph-mon[84247]: pgmap v146: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T14:01:56.425 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:56 smithi077 ceph-mon[84247]: osdmap e130: 8 total, 7 up, 8 in 2024-05-06T14:01:56.425 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:56 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:56.425 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:56 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:01:56.426 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:56 smithi077 ceph-mon[89212]: pgmap v146: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T14:01:56.426 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:56 smithi077 ceph-mon[89212]: osdmap e130: 8 total, 7 up, 8 in 2024-05-06T14:01:56.426 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:56 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:56.426 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:56 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:01:56.426 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:56 smithi077 podman[114152]: 2024-05-06 14:01:56.07942333 +0000 UTC m=+0.617012433 container died 2b6dce514a6796a7db6b89b496a8a6db2edf5cc21f0e177e16d099f857e243a1 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-70ce953, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-05-06T14:01:56.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:56 smithi170 ceph-mon[65866]: pgmap v146: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 149 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T14:01:56.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:56 smithi170 ceph-mon[65866]: osdmap e130: 8 total, 7 up, 8 in 2024-05-06T14:01:56.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:56 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:01:56.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:56 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:01:57.051 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:56 smithi077 podman[114178]: 2024-05-06 14:01:56.71186159 +0000 UTC m=+0.625027078 container remove 2b6dce514a6796a7db6b89b496a8a6db2edf5cc21f0e177e16d099f857e243a1 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-05-06T14:01:57.051 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:56 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.3.service: Deactivated successfully. 2024-05-06T14:01:57.051 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:56 smithi077 systemd[1]: Stopped Ceph osd.3 for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T14:01:57.052 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:56 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.3.service: Consumed 16.920s CPU time. 2024-05-06T14:01:57.052 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:56 smithi077 systemd[1]: Starting Ceph osd.3 for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:01:57.447 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:57 smithi077 ceph-mon[84247]: pgmap v148: 161 pgs: 13 active+undersized+degraded, 11 stale+active+clean, 26 active+undersized, 111 active+clean; 585 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 39/627 objects degraded (6.220%) 2024-05-06T14:01:57.448 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:01:57 smithi077 ceph-mon[84247]: Health check failed: Degraded data redundancy: 39/627 objects degraded (6.220%), 13 pgs degraded (PG_DEGRADED) 2024-05-06T14:01:57.448 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:57 smithi077 ceph-mon[89212]: pgmap v148: 161 pgs: 13 active+undersized+degraded, 11 stale+active+clean, 26 active+undersized, 111 active+clean; 585 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 39/627 objects degraded (6.220%) 2024-05-06T14:01:57.448 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:01:57 smithi077 ceph-mon[89212]: Health check failed: Degraded data redundancy: 39/627 objects degraded (6.220%), 13 pgs degraded (PG_DEGRADED) 2024-05-06T14:01:57.448 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:57 smithi077 podman[114281]: 2024-05-06 14:01:57.055010504 +0000 UTC m=+0.017961065 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:01:57.448 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:57 smithi077 podman[114281]: 2024-05-06 14:01:57.177631307 +0000 UTC m=+0.140581863 container create 530c78f6076349751eee59fd14f051be3cdd187a3987b69f5d20e165c8f722e9 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-70ce953) 2024-05-06T14:01:57.448 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:57 smithi077 podman[114281]: 2024-05-06 14:01:57.390895406 +0000 UTC m=+0.353845953 container init 530c78f6076349751eee59fd14f051be3cdd187a3987b69f5d20e165c8f722e9 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3-activate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-06T14:01:57.448 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:57 smithi077 podman[114281]: 2024-05-06 14:01:57.394768613 +0000 UTC m=+0.357719161 container start 530c78f6076349751eee59fd14f051be3cdd187a3987b69f5d20e165c8f722e9 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3-activate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-06T14:01:57.449 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:57 smithi077 podman[114281]: 2024-05-06 14:01:57.449029707 +0000 UTC m=+0.411980270 container attach 530c78f6076349751eee59fd14f051be3cdd187a3987b69f5d20e165c8f722e9 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3-activate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5) 2024-05-06T14:01:57.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:57 smithi170 ceph-mon[65866]: pgmap v148: 161 pgs: 13 active+undersized+degraded, 11 stale+active+clean, 26 active+undersized, 111 active+clean; 585 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 39/627 objects degraded (6.220%) 2024-05-06T14:01:57.664 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:01:57 smithi170 ceph-mon[65866]: Health check failed: Degraded data redundancy: 39/627 objects degraded (6.220%), 13 pgs degraded (PG_DEGRADED) 2024-05-06T14:01:58.126 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:57 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3-activate[114296]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-06T14:01:58.126 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:57 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3-activate[114296]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-05-06T14:01:58.126 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:57 smithi077 bash[114281]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-06T14:01:58.126 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:57 smithi077 bash[114281]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-05-06T14:01:58.126 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:57 smithi077 bash[114281]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-05-06T14:01:58.126 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:57 smithi077 bash[114281]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-05-06T14:01:58.126 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:57 smithi077 bash[114281]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-05-06T14:01:58.126 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:57 smithi077 bash[114281]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-06T14:01:58.126 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:57 smithi077 bash[114281]: --> ceph-volume raw activate successful for osd ID: 3 2024-05-06T14:01:58.127 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:57 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3-activate[114296]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-05-06T14:01:58.127 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:57 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3-activate[114296]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-05-06T14:01:58.127 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:57 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3-activate[114296]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-05-06T14:01:58.127 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:57 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3-activate[114296]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-06T14:01:58.127 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:57 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3-activate[114296]: --> ceph-volume raw activate successful for osd ID: 3 2024-05-06T14:01:58.127 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:57 smithi077 podman[114281]: 2024-05-06 14:01:57.858159259 +0000 UTC m=+0.821109806 container died 530c78f6076349751eee59fd14f051be3cdd187a3987b69f5d20e165c8f722e9 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3-activate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-06T14:01:58.551 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:58 smithi077 podman[114427]: 2024-05-06 14:01:58.352644323 +0000 UTC m=+0.489339312 container remove 530c78f6076349751eee59fd14f051be3cdd187a3987b69f5d20e165c8f722e9 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3-activate, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T14:01:58.938 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:58 smithi077 podman[114492]: 2024-05-06 14:01:58.561757895 +0000 UTC m=+0.112244786 container create 7c9e6dbacb8830e20649677a448f1e6709c4d3fffe0d16cd8d79a0508e0a75a5 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, ceph=True, CEPH_POINT_RELEASE=) 2024-05-06T14:01:58.938 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:58 smithi077 podman[114492]: 2024-05-06 14:01:58.467264654 +0000 UTC m=+0.017751573 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:01:59.301 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:58 smithi077 podman[114492]: 2024-05-06 14:01:58.939592072 +0000 UTC m=+0.490078973 container init 7c9e6dbacb8830e20649677a448f1e6709c4d3fffe0d16cd8d79a0508e0a75a5 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430) 2024-05-06T14:01:59.301 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:58 smithi077 podman[114492]: 2024-05-06 14:01:58.942948501 +0000 UTC m=+0.493435399 container start 7c9e6dbacb8830e20649677a448f1e6709c4d3fffe0d16cd8d79a0508e0a75a5 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-70ce953, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T14:01:59.301 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:58 smithi077 ceph-osd[114511]: -- 172.21.15.77:0/2730986992 <== mon.1 v2:172.21.15.77:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564292763680 con 0x56429272d400 2024-05-06T14:01:59.301 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:59 smithi077 bash[114492]: 7c9e6dbacb8830e20649677a448f1e6709c4d3fffe0d16cd8d79a0508e0a75a5 2024-05-06T14:01:59.302 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:59 smithi077 systemd[1]: Started Ceph osd.3 for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T14:01:59.801 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:01:59 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3[114507]: 2024-05-06T14:01:59.483+0000 7f8abeac77c0 -1 Falling back to public interface 2024-05-06T14:02:00.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:00 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:00.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:00 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:00.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:00 smithi077 ceph-mon[84247]: pgmap v149: 161 pgs: 19 active+undersized+degraded, 6 stale+active+clean, 33 active+undersized, 103 active+clean; 585 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 58/627 objects degraded (9.250%) 2024-05-06T14:02:00.301 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:00 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:00.301 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:00 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:00.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:00 smithi077 ceph-mon[89212]: pgmap v149: 161 pgs: 19 active+undersized+degraded, 6 stale+active+clean, 33 active+undersized, 103 active+clean; 585 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 58/627 objects degraded (9.250%) 2024-05-06T14:02:00.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:00 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:00.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:00 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:00.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:00 smithi170 ceph-mon[65866]: pgmap v149: 161 pgs: 19 active+undersized+degraded, 6 stale+active+clean, 33 active+undersized, 103 active+clean; 585 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 58/627 objects degraded (9.250%) 2024-05-06T14:02:02.481 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:02 smithi077 ceph-mon[84247]: pgmap v150: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 81/627 objects degraded (12.919%) 2024-05-06T14:02:02.481 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:02 smithi077 ceph-mon[89212]: pgmap v150: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 81/627 objects degraded (12.919%) 2024-05-06T14:02:02.481 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:02:02 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3[114507]: 2024-05-06T14:02:02.181+0000 7f8abeac77c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-05-06T14:02:02.481 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:02:02 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3[114507]: 2024-05-06T14:02:02.212+0000 7f8abeac77c0 -1 osd.3 128 log_to_monitors true 2024-05-06T14:02:02.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:02 smithi170 ceph-mon[65866]: pgmap v150: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 81/627 objects degraded (12.919%) 2024-05-06T14:02:03.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:03 smithi077 ceph-mon[84247]: from='osd.3 [v2:172.21.15.77:6826/3675133463,v1:172.21.15.77:6827/3675133463]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-06T14:02:03.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:03 smithi077 ceph-mon[84247]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-06T14:02:03.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:03 smithi077 ceph-mon[84247]: Health check update: Degraded data redundancy: 81/627 objects degraded (12.919%), 25 pgs degraded (PG_DEGRADED) 2024-05-06T14:02:03.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:03 smithi077 ceph-mon[89212]: from='osd.3 [v2:172.21.15.77:6826/3675133463,v1:172.21.15.77:6827/3675133463]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-06T14:02:03.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:03 smithi077 ceph-mon[89212]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-06T14:02:03.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:03 smithi077 ceph-mon[89212]: Health check update: Degraded data redundancy: 81/627 objects degraded (12.919%), 25 pgs degraded (PG_DEGRADED) 2024-05-06T14:02:03.552 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:02:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3[114507]: 2024-05-06T14:02:03.229+0000 7f8aaff1e700 -1 osd.3 128 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T14:02:03.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:03 smithi170 ceph-mon[65866]: from='osd.3 [v2:172.21.15.77:6826/3675133463,v1:172.21.15.77:6827/3675133463]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-06T14:02:03.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:03 smithi170 ceph-mon[65866]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-06T14:02:03.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:03 smithi170 ceph-mon[65866]: Health check update: Degraded data redundancy: 81/627 objects degraded (12.919%), 25 pgs degraded (PG_DEGRADED) 2024-05-06T14:02:04.051 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:02:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:02:03] "GET /metrics HTTP/1.1" 200 38187 "" "Prometheus/2.43.0" 2024-05-06T14:02:04.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:04 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:02:04.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:04 smithi077 ceph-mon[84247]: pgmap v151: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 81/627 objects degraded (12.919%) 2024-05-06T14:02:04.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:04 smithi077 ceph-mon[84247]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-06T14:02:04.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:04 smithi077 ceph-mon[84247]: from='osd.3 [v2:172.21.15.77:6826/3675133463,v1:172.21.15.77:6827/3675133463]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T14:02:04.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:04 smithi077 ceph-mon[84247]: osdmap e131: 8 total, 7 up, 8 in 2024-05-06T14:02:04.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:04 smithi077 ceph-mon[84247]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T14:02:04.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:04 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:04.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:04 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:04.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:04 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:02:04.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:04 smithi077 ceph-mon[89212]: pgmap v151: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 81/627 objects degraded (12.919%) 2024-05-06T14:02:04.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:04 smithi077 ceph-mon[89212]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-06T14:02:04.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:04 smithi077 ceph-mon[89212]: from='osd.3 [v2:172.21.15.77:6826/3675133463,v1:172.21.15.77:6827/3675133463]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T14:02:04.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:04 smithi077 ceph-mon[89212]: osdmap e131: 8 total, 7 up, 8 in 2024-05-06T14:02:04.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:04 smithi077 ceph-mon[89212]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T14:02:04.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:04 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:04.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:04 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:04.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:04 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:02:04.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:04 smithi170 ceph-mon[65866]: pgmap v151: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 150 MiB used, 715 GiB / 715 GiB avail; 81/627 objects degraded (12.919%) 2024-05-06T14:02:04.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:04 smithi170 ceph-mon[65866]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-06T14:02:04.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:04 smithi170 ceph-mon[65866]: from='osd.3 [v2:172.21.15.77:6826/3675133463,v1:172.21.15.77:6827/3675133463]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T14:02:04.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:04 smithi170 ceph-mon[65866]: osdmap e131: 8 total, 7 up, 8 in 2024-05-06T14:02:04.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:04 smithi170 ceph-mon[65866]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-05-06T14:02:04.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:04 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:04.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:04 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:05.461 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:05 smithi077 ceph-mon[89212]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T14:02:05.461 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:05 smithi077 ceph-mon[89212]: osd.3 [v2:172.21.15.77:6826/3675133463,v1:172.21.15.77:6827/3675133463] boot 2024-05-06T14:02:05.461 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:05 smithi077 ceph-mon[89212]: osdmap e132: 8 total, 8 up, 8 in 2024-05-06T14:02:05.461 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:05 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T14:02:05.461 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:05 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:05.461 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:05 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:05.461 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:05 smithi077 ceph-mon[89212]: pgmap v154: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 81/627 objects degraded (12.919%) 2024-05-06T14:02:05.461 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:05 smithi077 ceph-mon[89212]: osdmap e133: 8 total, 8 up, 8 in 2024-05-06T14:02:05.462 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:05 smithi077 ceph-mon[84247]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T14:02:05.462 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:05 smithi077 ceph-mon[84247]: osd.3 [v2:172.21.15.77:6826/3675133463,v1:172.21.15.77:6827/3675133463] boot 2024-05-06T14:02:05.462 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:05 smithi077 ceph-mon[84247]: osdmap e132: 8 total, 8 up, 8 in 2024-05-06T14:02:05.462 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:05 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T14:02:05.462 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:05 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:05.462 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:05 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:05.462 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:05 smithi077 ceph-mon[84247]: pgmap v154: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 81/627 objects degraded (12.919%) 2024-05-06T14:02:05.462 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:05 smithi077 ceph-mon[84247]: osdmap e133: 8 total, 8 up, 8 in 2024-05-06T14:02:05.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:05 smithi170 ceph-mon[65866]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T14:02:05.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:05 smithi170 ceph-mon[65866]: osd.3 [v2:172.21.15.77:6826/3675133463,v1:172.21.15.77:6827/3675133463] boot 2024-05-06T14:02:05.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:05 smithi170 ceph-mon[65866]: osdmap e132: 8 total, 8 up, 8 in 2024-05-06T14:02:05.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:05 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-06T14:02:05.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:05 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:05.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:05 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:05.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:05 smithi170 ceph-mon[65866]: pgmap v154: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 165 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 81/627 objects degraded (12.919%) 2024-05-06T14:02:05.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:05 smithi170 ceph-mon[65866]: osdmap e133: 8 total, 8 up, 8 in 2024-05-06T14:02:08.187 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:02:08 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T14:02:08.133+0000 7f5cd1795700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-05-06T14:02:08.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:08 smithi077 ceph-mon[89212]: pgmap v156: 161 pgs: 18 active+undersized+degraded, 25 active+undersized, 118 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 61/627 objects degraded (9.729%) 2024-05-06T14:02:08.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:08 smithi077 ceph-mon[89212]: Health check update: Degraded data redundancy: 61/627 objects degraded (9.729%), 18 pgs degraded (PG_DEGRADED) 2024-05-06T14:02:08.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:08 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:08.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:08 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:08.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:08 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:02:08.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:08 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:02:08.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:08 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:08.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:08 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:02:08.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:08 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:02:08.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:08 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:02:08.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:08 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:02:08.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:08 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-06T14:02:08.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:08 smithi077 ceph-mon[84247]: pgmap v156: 161 pgs: 18 active+undersized+degraded, 25 active+undersized, 118 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 61/627 objects degraded (9.729%) 2024-05-06T14:02:08.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:08 smithi077 ceph-mon[84247]: Health check update: Degraded data redundancy: 61/627 objects degraded (9.729%), 18 pgs degraded (PG_DEGRADED) 2024-05-06T14:02:08.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:08 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:08.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:08 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:08.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:08 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:02:08.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:08 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:02:08.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:08 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:08.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:08 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:02:08.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:08 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:02:08.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:08 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:02:08.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:08 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:02:08.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:08 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-06T14:02:08.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:08 smithi170 ceph-mon[65866]: pgmap v156: 161 pgs: 18 active+undersized+degraded, 25 active+undersized, 118 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 61/627 objects degraded (9.729%) 2024-05-06T14:02:08.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:08 smithi170 ceph-mon[65866]: Health check update: Degraded data redundancy: 61/627 objects degraded (9.729%), 18 pgs degraded (PG_DEGRADED) 2024-05-06T14:02:08.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:08 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:08.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:08 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:08.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:08 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:02:08.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:08 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:02:08.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:08 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:08.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:08 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:02:08.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:08 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:02:08.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:08 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:02:08.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:08 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:02:08.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:08 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-06T14:02:09.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:09 smithi077 ceph-mon[84247]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-06T14:02:09.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:09 smithi077 ceph-mon[84247]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-05-06T14:02:09.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:09 smithi077 ceph-mon[89212]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-06T14:02:09.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:09 smithi077 ceph-mon[89212]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-05-06T14:02:09.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:09 smithi170 ceph-mon[65866]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-06T14:02:09.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:09 smithi170 ceph-mon[65866]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-05-06T14:02:10.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:10 smithi077 ceph-mon[84247]: pgmap v157: 161 pgs: 4 active+undersized+degraded, 4 active+undersized, 153 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 15/627 objects degraded (2.392%) 2024-05-06T14:02:10.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:10 smithi077 ceph-mon[89212]: pgmap v157: 161 pgs: 4 active+undersized+degraded, 4 active+undersized, 153 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 15/627 objects degraded (2.392%) 2024-05-06T14:02:10.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:10 smithi170 ceph-mon[65866]: pgmap v157: 161 pgs: 4 active+undersized+degraded, 4 active+undersized, 153 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 15/627 objects degraded (2.392%) 2024-05-06T14:02:12.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:12 smithi077 ceph-mon[84247]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail 2024-05-06T14:02:12.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:12 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:12.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:12 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:02:12.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:12 smithi077 ceph-mon[84247]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 15/627 objects degraded (2.392%), 4 pgs degraded) 2024-05-06T14:02:12.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:12 smithi077 ceph-mon[84247]: Cluster is now healthy 2024-05-06T14:02:12.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:12 smithi077 ceph-mon[89212]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail 2024-05-06T14:02:12.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:12 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:12.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:12 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:02:12.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:12 smithi077 ceph-mon[89212]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 15/627 objects degraded (2.392%), 4 pgs degraded) 2024-05-06T14:02:12.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:12 smithi077 ceph-mon[89212]: Cluster is now healthy 2024-05-06T14:02:12.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:12 smithi170 ceph-mon[65866]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail 2024-05-06T14:02:12.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:12 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:12.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:12 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:02:12.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:12 smithi170 ceph-mon[65866]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 15/627 objects degraded (2.392%), 4 pgs degraded) 2024-05-06T14:02:12.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:12 smithi170 ceph-mon[65866]: Cluster is now healthy 2024-05-06T14:02:14.051 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:02:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:02:13] "GET /metrics HTTP/1.1" 200 38183 "" "Prometheus/2.43.0" 2024-05-06T14:02:14.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:14 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:02:14.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:14 smithi077 ceph-mon[84247]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail 2024-05-06T14:02:14.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:14 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:02:14.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:14 smithi077 ceph-mon[89212]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail 2024-05-06T14:02:14.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:14 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:02:14.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:14 smithi170 ceph-mon[65866]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail 2024-05-06T14:02:16.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:16 smithi077 ceph-mon[84247]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-05-06T14:02:16.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:16 smithi077 ceph-mon[89212]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-05-06T14:02:16.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:16 smithi170 ceph-mon[65866]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-05-06T14:02:17.801 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:17 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:17.801 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:17 smithi077 ceph-mon[84247]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-05-06T14:02:17.801 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:17 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:17.801 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:17 smithi077 ceph-mon[89212]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-05-06T14:02:17.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:17 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:17.825 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:17 smithi170 ceph-mon[65866]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-05-06T14:02:20.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:20 smithi077 ceph-mon[84247]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-06T14:02:20.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:20 smithi077 ceph-mon[89212]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-06T14:02:20.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:20 smithi170 ceph-mon[65866]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-05-06T14:02:21.359 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T14:02:22.008 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T14:02:22.008 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (7m) 18s ago 29m 26.6M - 0.25.0 c8568f914cd2 997175e3e64e 2024-05-06T14:02:22.008 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (6m) 4m ago 29m 80.6M - 9.4.12 2bacad6d85d8 140bb9a2760a 2024-05-06T14:02:22.008 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (7m) 18s ago 28m 64.2M - 3.5 e1d6a67b021e 9353a303fb55 2024-05-06T14:02:22.009 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283,8765 running (8m) 4m ago 34m 447M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 125521be53a3 2024-05-06T14:02:22.009 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (18m) 18s ago 35m 509M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T14:02:22.009 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (4m) 18s ago 35m 50.5M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 5e50731ccbe8 2024-05-06T14:02:22.009 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (5m) 4m ago 34m 36.3M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 61cd48b3d0ff 2024-05-06T14:02:22.009 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (4m) 18s ago 34m 40.9M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 8a473cc2ffd1 2024-05-06T14:02:22.009 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (8m) 18s ago 30m 15.9M - 1.5.0 0da6a335fe13 67f45fa4968b 2024-05-06T14:02:22.009 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (8m) 4m ago 30m 16.4M - 1.5.0 0da6a335fe13 02081b71a5dd 2024-05-06T14:02:22.009 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (2m) 18s ago 33m 43.8M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 fbe876a933ed 2024-05-06T14:02:22.009 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (2m) 18s ago 33m 44.5M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 6f664e567c93 2024-05-06T14:02:22.009 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (76s) 18s ago 32m 43.8M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 a4c3266b87f8 2024-05-06T14:02:22.009 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (23s) 18s ago 32m 12.6M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 7c9e6dbacb88 2024-05-06T14:02:22.009 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (32m) 4m ago 32m 58.5M 3248M 17.2.0 e1d6a67b021e f865acc44a06 2024-05-06T14:02:22.009 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (31m) 4m ago 31m 52.8M 3248M 17.2.0 e1d6a67b021e a55bd4122f6f 2024-05-06T14:02:22.010 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (31m) 4m ago 31m 51.4M 3248M 17.2.0 e1d6a67b021e 362938a0b357 2024-05-06T14:02:22.010 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (31m) 4m ago 31m 54.1M 3248M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T14:02:22.010 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (7m) 4m ago 30m 60.1M - 2.43.0 a07b618ecd1d 10d5c513bb9b 2024-05-06T14:02:22.010 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (28m) 18s ago 28m 79.4M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T14:02:22.010 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (28m) 4m ago 28m 77.2M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T14:02:22.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:22 smithi077 ceph-mon[84247]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:02:22.301 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:22 smithi077 ceph-mon[89212]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:02:22.405 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:02:22.405 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-05-06T14:02:22.405 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T14:02:22.405 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:02:22.405 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-05-06T14:02:22.405 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T14:02:22.405 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:02:22.405 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-05-06T14:02:22.406 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-05-06T14:02:22.406 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 4 2024-05-06T14:02:22.406 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:02:22.406 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw": { 2024-05-06T14:02:22.406 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T14:02:22.406 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:02:22.406 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-05-06T14:02:22.406 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-05-06T14:02:22.406 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 9 2024-05-06T14:02:22.406 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T14:02:22.406 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:02:22.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:22 smithi170 ceph-mon[65866]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:02:22.735 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:02:22.736 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T14:02:22.736 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-05-06T14:02:22.736 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-05-06T14:02:22.736 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [], 2024-05-06T14:02:22.736 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "4/8 daemons upgraded", 2024-05-06T14:02:22.736 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "Currently upgrading osd daemons", 2024-05-06T14:02:22.736 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-05-06T14:02:22.736 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:02:23.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:23 smithi077 ceph-mon[89212]: from='client.64408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:02:23.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:23 smithi077 ceph-mon[89212]: from='client.54480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:02:23.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:23 smithi077 ceph-mon[89212]: from='client.54486 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:02:23.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:23 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/2969475741' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:02:23.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:23 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-06T14:02:23.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:23 smithi077 ceph-mon[84247]: from='client.64408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:02:23.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:23 smithi077 ceph-mon[84247]: from='client.54480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:02:23.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:23 smithi077 ceph-mon[84247]: from='client.54486 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:02:23.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:23 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/2969475741' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:02:23.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:23 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-06T14:02:23.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:23 smithi170 ceph-mon[65866]: from='client.64408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:02:23.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:23 smithi170 ceph-mon[65866]: from='client.54480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:02:23.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:23 smithi170 ceph-mon[65866]: from='client.54486 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:02:23.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:23 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/2969475741' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:02:23.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:23 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-06T14:02:24.050 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:02:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:02:23] "GET /metrics HTTP/1.1" 200 38183 "" "Prometheus/2.43.0" 2024-05-06T14:02:24.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:24 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:02:24.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:24 smithi170 ceph-mon[65866]: from='client.54498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:02:24.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:24 smithi170 ceph-mon[65866]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-06T14:02:24.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:24 smithi170 ceph-mon[65866]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:02:24.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:24 smithi170 ceph-mon[65866]: Upgrade: osd.4 is safe to restart 2024-05-06T14:02:24.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:24 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:02:24.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:24 smithi077 ceph-mon[89212]: from='client.54498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:02:24.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:24 smithi077 ceph-mon[89212]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-06T14:02:24.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:24 smithi077 ceph-mon[89212]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:02:24.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:24 smithi077 ceph-mon[89212]: Upgrade: osd.4 is safe to restart 2024-05-06T14:02:24.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:24 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:02:24.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:24 smithi077 ceph-mon[84247]: from='client.54498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:02:24.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:24 smithi077 ceph-mon[84247]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-05-06T14:02:24.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:24 smithi077 ceph-mon[84247]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:02:24.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:24 smithi077 ceph-mon[84247]: Upgrade: osd.4 is safe to restart 2024-05-06T14:02:26.173 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:25 smithi170 ceph-mon[65866]: Upgrade: Updating osd.4 2024-05-06T14:02:26.173 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:26.173 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-06T14:02:26.173 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:25 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:02:26.173 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:25 smithi170 ceph-mon[65866]: Deploying daemon osd.4 on smithi170 2024-05-06T14:02:26.173 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:25 smithi170 ceph-mon[65866]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:02:26.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:25 smithi077 ceph-mon[84247]: Upgrade: Updating osd.4 2024-05-06T14:02:26.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:26.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-06T14:02:26.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:25 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:02:26.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:25 smithi077 ceph-mon[84247]: Deploying daemon osd.4 on smithi170 2024-05-06T14:02:26.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:25 smithi077 ceph-mon[84247]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:02:26.301 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:25 smithi077 ceph-mon[89212]: Upgrade: Updating osd.4 2024-05-06T14:02:26.301 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:26.301 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-06T14:02:26.301 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:25 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:02:26.301 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:25 smithi077 ceph-mon[89212]: Deploying daemon osd.4 on smithi170 2024-05-06T14:02:26.301 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:25 smithi077 ceph-mon[89212]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:02:27.482 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:27 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:27.482 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:27 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:02:27.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:27 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:27.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:27 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:02:27.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:27 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:27.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:27 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:02:28.446 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:28 smithi170 ceph-mon[65866]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:02:28.446 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:28 smithi170 systemd[1]: Stopping Ceph osd.4 for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:02:28.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:28 smithi077 ceph-mon[84247]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:02:28.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:28 smithi077 ceph-mon[89212]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:02:28.825 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4[26640]: 2024-05-06T14:02:28.446+0000 7f767f95c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:02:28.825 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4[26640]: 2024-05-06T14:02:28.446+0000 7f767f95c700 -1 osd.4 133 *** Got signal Terminated *** 2024-05-06T14:02:28.825 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4[26640]: 2024-05-06T14:02:28.446+0000 7f767f95c700 -1 osd.4 133 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T14:02:29.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:29 smithi077 ceph-mon[84247]: osd.4 marked itself down and dead 2024-05-06T14:02:29.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:29 smithi077 ceph-mon[89212]: osd.4 marked itself down and dead 2024-05-06T14:02:29.568 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:29 smithi170 ceph-mon[65866]: osd.4 marked itself down and dead 2024-05-06T14:02:29.824 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:29 smithi170 podman[75576]: 2024-05-06 14:02:29.569423048 +0000 UTC m=+1.237244657 container died f865acc44a06d90e98c6176c30302c08cb36084ea5c1083f4a8781979cc4bd50 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, version=8, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, vendor=Red Hat, Inc., io.openshift.expose-services=, ceph=True, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-05-06T14:02:30.154 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:29 smithi170 podman[75576]: 2024-05-06 14:02:29.922065651 +0000 UTC m=+1.589887267 container cleanup f865acc44a06d90e98c6176c30302c08cb36084ea5c1083f4a8781979cc4bd50 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, release=754, io.buildah.version=1.19.8, io.openshift.expose-services=, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, ceph=True, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , distribution-scope=public) 2024-05-06T14:02:30.154 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:29 smithi170 bash[75576]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4 2024-05-06T14:02:30.539 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:30 smithi170 podman[75593]: 2024-05-06 14:02:30.207417979 +0000 UTC m=+0.637348729 container remove f865acc44a06d90e98c6176c30302c08cb36084ea5c1083f4a8781979cc4bd50 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, ceph=True, version=8, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, architecture=x86_64) 2024-05-06T14:02:30.540 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:30 smithi170 ceph-mon[65866]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:02:30.540 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:30 smithi170 ceph-mon[65866]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T14:02:30.540 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:30 smithi170 ceph-mon[65866]: osdmap e134: 8 total, 7 up, 8 in 2024-05-06T14:02:30.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:30 smithi077 ceph-mon[84247]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:02:30.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:30 smithi077 ceph-mon[84247]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T14:02:30.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:30 smithi077 ceph-mon[84247]: osdmap e134: 8 total, 7 up, 8 in 2024-05-06T14:02:30.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:30 smithi077 ceph-mon[89212]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:02:30.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:30 smithi077 ceph-mon[89212]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T14:02:30.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:30 smithi077 ceph-mon[89212]: osdmap e134: 8 total, 7 up, 8 in 2024-05-06T14:02:30.809 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:30 smithi170 podman[75686]: 2024-05-06 14:02:30.444251809 +0000 UTC m=+0.018452308 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:02:30.809 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:30 smithi170 podman[75686]: 2024-05-06 14:02:30.541612503 +0000 UTC m=+0.115813001 container create 55840f31f44a2b5bee658a64f9942882ae46ef7fa0916a0e16de5ebb41075ab8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-70ce953, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-05-06T14:02:30.809 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:30 smithi170 podman[75686]: 2024-05-06 14:02:30.728070788 +0000 UTC m=+0.302271286 container init 55840f31f44a2b5bee658a64f9942882ae46ef7fa0916a0e16de5ebb41075ab8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430) 2024-05-06T14:02:30.809 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:30 smithi170 podman[75686]: 2024-05-06 14:02:30.731314752 +0000 UTC m=+0.305515250 container start 55840f31f44a2b5bee658a64f9942882ae46ef7fa0916a0e16de5ebb41075ab8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-70ce953, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-06T14:02:31.074 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:30 smithi170 podman[75686]: 2024-05-06 14:02:30.810873726 +0000 UTC m=+0.385074233 container attach 55840f31f44a2b5bee658a64f9942882ae46ef7fa0916a0e16de5ebb41075ab8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-06T14:02:31.074 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:31 smithi170 podman[75686]: 2024-05-06 14:02:31.023840956 +0000 UTC m=+0.598041444 container died 55840f31f44a2b5bee658a64f9942882ae46ef7fa0916a0e16de5ebb41075ab8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-70ce953, io.buildah.version=1.33.5, org.label-schema.build-date=20240430) 2024-05-06T14:02:31.550 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:31 smithi170 ceph-mon[65866]: osdmap e135: 8 total, 7 up, 8 in 2024-05-06T14:02:31.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:31 smithi077 ceph-mon[84247]: osdmap e135: 8 total, 7 up, 8 in 2024-05-06T14:02:31.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:31 smithi077 ceph-mon[89212]: osdmap e135: 8 total, 7 up, 8 in 2024-05-06T14:02:31.806 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:31 smithi170 podman[75713]: 2024-05-06 14:02:31.552173297 +0000 UTC m=+0.522472387 container remove 55840f31f44a2b5bee658a64f9942882ae46ef7fa0916a0e16de5ebb41075ab8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T14:02:31.807 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:31 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.4.service: Deactivated successfully. 2024-05-06T14:02:31.807 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:31 smithi170 systemd[1]: Stopped Ceph osd.4 for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T14:02:31.807 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:31 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.4.service: Consumed 16.902s CPU time. 2024-05-06T14:02:31.807 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:31 smithi170 systemd[1]: Starting Ceph osd.4 for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:02:32.074 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:32 smithi170 podman[75821]: 2024-05-06 14:02:32.024637296 +0000 UTC m=+0.114484742 container create 446da0b13460c13cacd7069363c7e2ada84e3ebfab3ec0cae51644afee65409b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4-activate, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, org.label-schema.build-date=20240430) 2024-05-06T14:02:32.075 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:32 smithi170 podman[75821]: 2024-05-06 14:02:31.928660577 +0000 UTC m=+0.018507992 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:02:32.452 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:32 smithi077 ceph-mon[84247]: pgmap v170: 161 pgs: 2 active+undersized+degraded, 9 peering, 18 stale+active+clean, 4 active+undersized, 128 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 6/627 objects degraded (0.957%) 2024-05-06T14:02:32.452 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:32 smithi077 ceph-mon[84247]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-05-06T14:02:32.452 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:32 smithi077 ceph-mon[84247]: Health check failed: Degraded data redundancy: 6/627 objects degraded (0.957%), 2 pgs degraded (PG_DEGRADED) 2024-05-06T14:02:32.452 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:32 smithi077 ceph-mon[89212]: pgmap v170: 161 pgs: 2 active+undersized+degraded, 9 peering, 18 stale+active+clean, 4 active+undersized, 128 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 6/627 objects degraded (0.957%) 2024-05-06T14:02:32.453 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:32 smithi077 ceph-mon[89212]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-05-06T14:02:32.453 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:32 smithi077 ceph-mon[89212]: Health check failed: Degraded data redundancy: 6/627 objects degraded (0.957%), 2 pgs degraded (PG_DEGRADED) 2024-05-06T14:02:32.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:32 smithi170 ceph-mon[65866]: pgmap v170: 161 pgs: 2 active+undersized+degraded, 9 peering, 18 stale+active+clean, 4 active+undersized, 128 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 6/627 objects degraded (0.957%) 2024-05-06T14:02:32.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:32 smithi170 ceph-mon[65866]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-05-06T14:02:32.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:32 smithi170 ceph-mon[65866]: Health check failed: Degraded data redundancy: 6/627 objects degraded (0.957%), 2 pgs degraded (PG_DEGRADED) 2024-05-06T14:02:32.575 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:32 smithi170 podman[75821]: 2024-05-06 14:02:32.252509514 +0000 UTC m=+0.342356932 container init 446da0b13460c13cacd7069363c7e2ada84e3ebfab3ec0cae51644afee65409b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4-activate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T14:02:32.575 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:32 smithi170 podman[75821]: 2024-05-06 14:02:32.256182271 +0000 UTC m=+0.346029672 container start 446da0b13460c13cacd7069363c7e2ada84e3ebfab3ec0cae51644afee65409b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4-activate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-05-06T14:02:32.575 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:32 smithi170 podman[75821]: 2024-05-06 14:02:32.369096525 +0000 UTC m=+0.458943931 container attach 446da0b13460c13cacd7069363c7e2ada84e3ebfab3ec0cae51644afee65409b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4-activate, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-70ce953, GIT_CLEAN=True) 2024-05-06T14:02:33.027 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:32 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4-activate[75835]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-06T14:02:33.027 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:32 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4-activate[75835]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-05-06T14:02:33.027 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:32 smithi170 bash[75821]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-06T14:02:33.027 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:32 smithi170 bash[75821]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-05-06T14:02:33.027 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:32 smithi170 bash[75821]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-05-06T14:02:33.027 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:32 smithi170 bash[75821]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-05-06T14:02:33.027 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:32 smithi170 bash[75821]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-05-06T14:02:33.028 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:32 smithi170 bash[75821]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-06T14:02:33.028 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:32 smithi170 bash[75821]: --> ceph-volume raw activate successful for osd ID: 4 2024-05-06T14:02:33.028 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:32 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4-activate[75835]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-05-06T14:02:33.028 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:32 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4-activate[75835]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-05-06T14:02:33.028 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:32 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4-activate[75835]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-05-06T14:02:33.028 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:32 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4-activate[75835]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-06T14:02:33.028 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:32 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4-activate[75835]: --> ceph-volume raw activate successful for osd ID: 4 2024-05-06T14:02:33.028 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:32 smithi170 podman[75821]: 2024-05-06 14:02:32.729220715 +0000 UTC m=+0.819068125 container died 446da0b13460c13cacd7069363c7e2ada84e3ebfab3ec0cae51644afee65409b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4-activate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240430, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T14:02:33.648 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:33 smithi170 podman[75963]: 2024-05-06 14:02:33.331902255 +0000 UTC m=+0.596695703 container remove 446da0b13460c13cacd7069363c7e2ada84e3ebfab3ec0cae51644afee65409b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4-activate, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-06T14:02:33.648 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:33 smithi170 podman[76029]: 2024-05-06 14:02:33.54145364 +0000 UTC m=+0.110527132 container create 345a6ce2c7c339d4812ff990f8f94c3ec5b5ad8a68105243c1aa74c4cf449093 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, ceph=True) 2024-05-06T14:02:33.649 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:33 smithi170 podman[76029]: 2024-05-06 14:02:33.449149751 +0000 UTC m=+0.018223209 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:02:34.051 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:02:33 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:02:33] "GET /metrics HTTP/1.1" 200 38182 "" "Prometheus/2.43.0" 2024-05-06T14:02:34.074 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:33 smithi170 podman[76029]: 2024-05-06 14:02:33.735440717 +0000 UTC m=+0.304514172 container init 345a6ce2c7c339d4812ff990f8f94c3ec5b5ad8a68105243c1aa74c4cf449093 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-06T14:02:34.074 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:33 smithi170 podman[76029]: 2024-05-06 14:02:33.738579268 +0000 UTC m=+0.307652717 container start 345a6ce2c7c339d4812ff990f8f94c3ec5b5ad8a68105243c1aa74c4cf449093 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T14:02:34.074 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:33 smithi170 ceph-osd[76048]: -- 172.21.15.170:0/1764121148 <== mon.2 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559fd6507680 con 0x559fd64d1000 2024-05-06T14:02:34.074 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:33 smithi170 bash[76029]: 345a6ce2c7c339d4812ff990f8f94c3ec5b5ad8a68105243c1aa74c4cf449093 2024-05-06T14:02:34.075 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:33 smithi170 systemd[1]: Started Ceph osd.4 for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T14:02:34.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:34 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:02:34.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:34 smithi077 ceph-mon[84247]: pgmap v171: 161 pgs: 15 active+undersized+degraded, 9 peering, 9 stale+active+clean, 23 active+undersized, 105 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 63/627 objects degraded (10.048%) 2024-05-06T14:02:34.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:34 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:34.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:34 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:34.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:34 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:02:34.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:34 smithi077 ceph-mon[89212]: pgmap v171: 161 pgs: 15 active+undersized+degraded, 9 peering, 9 stale+active+clean, 23 active+undersized, 105 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 63/627 objects degraded (10.048%) 2024-05-06T14:02:34.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:34 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:34.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:34 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:34.574 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:34 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4[76044]: 2024-05-06T14:02:34.294+0000 7fc85a5307c0 -1 Falling back to public interface 2024-05-06T14:02:34.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:34 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:02:34.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:34 smithi170 ceph-mon[65866]: pgmap v171: 161 pgs: 15 active+undersized+degraded, 9 peering, 9 stale+active+clean, 23 active+undersized, 105 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 63/627 objects degraded (10.048%) 2024-05-06T14:02:34.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:34 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:34.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:34 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:35.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:35 smithi077 ceph-mon[89212]: pgmap v172: 161 pgs: 28 active+undersized+degraded, 40 active+undersized, 93 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 108/627 objects degraded (17.225%) 2024-05-06T14:02:35.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:35 smithi077 ceph-mon[89212]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-05-06T14:02:35.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:35 smithi077 ceph-mon[84247]: pgmap v172: 161 pgs: 28 active+undersized+degraded, 40 active+undersized, 93 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 108/627 objects degraded (17.225%) 2024-05-06T14:02:35.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:35 smithi077 ceph-mon[84247]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-05-06T14:02:35.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:35 smithi170 ceph-mon[65866]: pgmap v172: 161 pgs: 28 active+undersized+degraded, 40 active+undersized, 93 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 108/627 objects degraded (17.225%) 2024-05-06T14:02:35.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:35 smithi170 ceph-mon[65866]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-05-06T14:02:37.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:37 smithi170 ceph-mon[65866]: from='osd.4 [v2:172.21.15.170:6800/4041656149,v1:172.21.15.170:6801/4041656149]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T14:02:37.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:37 smithi170 ceph-mon[65866]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T14:02:37.325 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:37 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4[76044]: 2024-05-06T14:02:37.000+0000 7fc85a5307c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-05-06T14:02:37.325 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:37 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4[76044]: 2024-05-06T14:02:37.031+0000 7fc85a5307c0 -1 osd.4 133 log_to_monitors true 2024-05-06T14:02:37.325 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:02:37 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4[76044]: 2024-05-06T14:02:37.116+0000 7fc84b987700 -1 osd.4 133 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T14:02:37.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:37 smithi077 ceph-mon[84247]: from='osd.4 [v2:172.21.15.170:6800/4041656149,v1:172.21.15.170:6801/4041656149]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T14:02:37.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:37 smithi077 ceph-mon[84247]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T14:02:37.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:37 smithi077 ceph-mon[89212]: from='osd.4 [v2:172.21.15.170:6800/4041656149,v1:172.21.15.170:6801/4041656149]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T14:02:37.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:37 smithi077 ceph-mon[89212]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-06T14:02:38.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:38 smithi077 ceph-mon[84247]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-06T14:02:38.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:38 smithi077 ceph-mon[84247]: from='osd.4 [v2:172.21.15.170:6800/4041656149,v1:172.21.15.170:6801/4041656149]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T14:02:38.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:38 smithi077 ceph-mon[84247]: osdmap e136: 8 total, 7 up, 8 in 2024-05-06T14:02:38.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:38 smithi077 ceph-mon[84247]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T14:02:38.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:38 smithi077 ceph-mon[84247]: pgmap v174: 161 pgs: 28 active+undersized+degraded, 40 active+undersized, 93 active+clean; 585 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 108/627 objects degraded (17.225%) 2024-05-06T14:02:38.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:38 smithi077 ceph-mon[84247]: Health check update: Degraded data redundancy: 108/627 objects degraded (17.225%), 28 pgs degraded (PG_DEGRADED) 2024-05-06T14:02:38.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:38 smithi077 ceph-mon[89212]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-06T14:02:38.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:38 smithi077 ceph-mon[89212]: from='osd.4 [v2:172.21.15.170:6800/4041656149,v1:172.21.15.170:6801/4041656149]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T14:02:38.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:38 smithi077 ceph-mon[89212]: osdmap e136: 8 total, 7 up, 8 in 2024-05-06T14:02:38.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:38 smithi077 ceph-mon[89212]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T14:02:38.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:38 smithi077 ceph-mon[89212]: pgmap v174: 161 pgs: 28 active+undersized+degraded, 40 active+undersized, 93 active+clean; 585 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 108/627 objects degraded (17.225%) 2024-05-06T14:02:38.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:38 smithi077 ceph-mon[89212]: Health check update: Degraded data redundancy: 108/627 objects degraded (17.225%), 28 pgs degraded (PG_DEGRADED) 2024-05-06T14:02:38.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:38 smithi170 ceph-mon[65866]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-06T14:02:38.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:38 smithi170 ceph-mon[65866]: from='osd.4 [v2:172.21.15.170:6800/4041656149,v1:172.21.15.170:6801/4041656149]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T14:02:38.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:38 smithi170 ceph-mon[65866]: osdmap e136: 8 total, 7 up, 8 in 2024-05-06T14:02:38.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:38 smithi170 ceph-mon[65866]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T14:02:38.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:38 smithi170 ceph-mon[65866]: pgmap v174: 161 pgs: 28 active+undersized+degraded, 40 active+undersized, 93 active+clean; 585 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 108/627 objects degraded (17.225%) 2024-05-06T14:02:38.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:38 smithi170 ceph-mon[65866]: Health check update: Degraded data redundancy: 108/627 objects degraded (17.225%), 28 pgs degraded (PG_DEGRADED) 2024-05-06T14:02:39.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:39 smithi077 ceph-mon[84247]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T14:02:39.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:39 smithi077 ceph-mon[84247]: osd.4 [v2:172.21.15.170:6800/4041656149,v1:172.21.15.170:6801/4041656149] boot 2024-05-06T14:02:39.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:39 smithi077 ceph-mon[84247]: osdmap e137: 8 total, 8 up, 8 in 2024-05-06T14:02:39.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:39 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T14:02:39.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:39 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:39.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:39 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:39.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:39 smithi077 ceph-mon[84247]: osdmap e138: 8 total, 8 up, 8 in 2024-05-06T14:02:39.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:39 smithi077 ceph-mon[89212]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T14:02:39.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:39 smithi077 ceph-mon[89212]: osd.4 [v2:172.21.15.170:6800/4041656149,v1:172.21.15.170:6801/4041656149] boot 2024-05-06T14:02:39.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:39 smithi077 ceph-mon[89212]: osdmap e137: 8 total, 8 up, 8 in 2024-05-06T14:02:39.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:39 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T14:02:39.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:39 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:39.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:39 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:39.552 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:39 smithi077 ceph-mon[89212]: osdmap e138: 8 total, 8 up, 8 in 2024-05-06T14:02:39.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:39 smithi170 ceph-mon[65866]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T14:02:39.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:39 smithi170 ceph-mon[65866]: osd.4 [v2:172.21.15.170:6800/4041656149,v1:172.21.15.170:6801/4041656149] boot 2024-05-06T14:02:39.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:39 smithi170 ceph-mon[65866]: osdmap e137: 8 total, 8 up, 8 in 2024-05-06T14:02:39.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:39 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-06T14:02:39.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:39 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:39.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:39 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:39.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:39 smithi170 ceph-mon[65866]: osdmap e138: 8 total, 8 up, 8 in 2024-05-06T14:02:40.265 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:40 smithi170 ceph-mon[65866]: pgmap v177: 161 pgs: 12 peering, 25 active+undersized+degraded, 31 active+undersized, 93 active+clean; 585 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 96/627 objects degraded (15.311%) 2024-05-06T14:02:40.265 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:40 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:40.265 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:40 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:40.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:40 smithi077 ceph-mon[84247]: pgmap v177: 161 pgs: 12 peering, 25 active+undersized+degraded, 31 active+undersized, 93 active+clean; 585 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 96/627 objects degraded (15.311%) 2024-05-06T14:02:40.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:40 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:40.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:40 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:40.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:40 smithi077 ceph-mon[89212]: pgmap v177: 161 pgs: 12 peering, 25 active+undersized+degraded, 31 active+undersized, 93 active+clean; 585 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 96/627 objects degraded (15.311%) 2024-05-06T14:02:40.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:40 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:40.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:40 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:42.301 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:42 smithi170 ceph-mon[65866]: pgmap v178: 161 pgs: 12 peering, 20 active+undersized+degraded, 26 active+undersized, 103 active+clean; 585 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 79/627 objects degraded (12.600%) 2024-05-06T14:02:42.301 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:42 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:42.302 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:42 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:02:42.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:42 smithi077 ceph-mon[89212]: pgmap v178: 161 pgs: 12 peering, 20 active+undersized+degraded, 26 active+undersized, 103 active+clean; 585 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 79/627 objects degraded (12.600%) 2024-05-06T14:02:42.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:42 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:42.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:42 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:02:42.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:42 smithi077 ceph-mon[84247]: pgmap v178: 161 pgs: 12 peering, 20 active+undersized+degraded, 26 active+undersized, 103 active+clean; 585 KiB data, 181 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 79/627 objects degraded (12.600%) 2024-05-06T14:02:42.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:42 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:42.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:42 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:02:43.115 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:02:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T14:02:42.865+0000 7f5cd1795700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-05-06T14:02:44.050 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:02:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:02:43] "GET /metrics HTTP/1.1" 200 38056 "" "Prometheus/2.43.0" 2024-05-06T14:02:44.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:02:44.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:44.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:44.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:02:44.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:02:44.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:44.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[84247]: Health check update: Degraded data redundancy: 79/627 objects degraded (12.600%), 20 pgs degraded (PG_DEGRADED) 2024-05-06T14:02:44.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:02:44.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:02:44.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:02:44.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:02:44.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-06T14:02:44.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[84247]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-06T14:02:44.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[84247]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-05-06T14:02:44.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[84247]: pgmap v179: 161 pgs: 12 peering, 149 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 845 B/s rd, 0 op/s 2024-05-06T14:02:44.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:02:44.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:44.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:44.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:02:44.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:02:44.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:44.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[89212]: Health check update: Degraded data redundancy: 79/627 objects degraded (12.600%), 20 pgs degraded (PG_DEGRADED) 2024-05-06T14:02:44.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:02:44.052 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:02:44.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:02:44.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:02:44.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-06T14:02:44.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[89212]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-06T14:02:44.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[89212]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-05-06T14:02:44.053 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:43 smithi077 ceph-mon[89212]: pgmap v179: 161 pgs: 12 peering, 149 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 845 B/s rd, 0 op/s 2024-05-06T14:02:44.065 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:43 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:02:44.065 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:43 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:44.065 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:43 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:44.065 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:43 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:02:44.065 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:43 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:02:44.065 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:43 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:44.065 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:43 smithi170 ceph-mon[65866]: Health check update: Degraded data redundancy: 79/627 objects degraded (12.600%), 20 pgs degraded (PG_DEGRADED) 2024-05-06T14:02:44.065 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:43 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:02:44.065 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:43 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:02:44.065 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:43 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:02:44.065 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:43 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:02:44.065 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:43 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-06T14:02:44.065 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:43 smithi170 ceph-mon[65866]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-06T14:02:44.065 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:43 smithi170 ceph-mon[65866]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-05-06T14:02:44.066 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:43 smithi170 ceph-mon[65866]: pgmap v179: 161 pgs: 12 peering, 149 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 845 B/s rd, 0 op/s 2024-05-06T14:02:44.967 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:44 smithi170 ceph-mon[65866]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 79/627 objects degraded (12.600%), 20 pgs degraded) 2024-05-06T14:02:44.967 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:44 smithi170 ceph-mon[65866]: Cluster is now healthy 2024-05-06T14:02:45.050 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:44 smithi077 ceph-mon[84247]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 79/627 objects degraded (12.600%), 20 pgs degraded) 2024-05-06T14:02:45.050 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:44 smithi077 ceph-mon[84247]: Cluster is now healthy 2024-05-06T14:02:45.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:44 smithi077 ceph-mon[89212]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 79/627 objects degraded (12.600%), 20 pgs degraded) 2024-05-06T14:02:45.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:44 smithi077 ceph-mon[89212]: Cluster is now healthy 2024-05-06T14:02:46.050 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:45 smithi077 ceph-mon[84247]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:02:46.050 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:45 smithi077 ceph-mon[89212]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:02:46.074 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:45 smithi170 ceph-mon[65866]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:02:48.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:48 smithi077 ceph-mon[84247]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T14:02:48.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:48 smithi077 ceph-mon[89212]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T14:02:48.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:48 smithi170 ceph-mon[65866]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T14:02:50.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:50 smithi077 ceph-mon[84247]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s 2024-05-06T14:02:50.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:50 smithi077 ceph-mon[89212]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s 2024-05-06T14:02:50.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:50 smithi170 ceph-mon[65866]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s 2024-05-06T14:02:52.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:52 smithi077 ceph-mon[84247]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:02:52.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:52 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:52.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:52 smithi077 ceph-mon[89212]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:02:52.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:52 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:52.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:52 smithi170 ceph-mon[65866]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:02:52.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:52 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:53.094 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T14:02:53.751 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T14:02:53.751 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (7m) 50s ago 30m 26.6M - 0.25.0 c8568f914cd2 997175e3e64e 2024-05-06T14:02:53.751 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (7m) 15s ago 29m 81.4M - 9.4.12 2bacad6d85d8 140bb9a2760a 2024-05-06T14:02:53.751 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (8m) 50s ago 29m 64.2M - 3.5 e1d6a67b021e 9353a303fb55 2024-05-06T14:02:53.751 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283,8765 running (9m) 15s ago 34m 452M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 125521be53a3 2024-05-06T14:02:53.752 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (18m) 50s ago 36m 509M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T14:02:53.752 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (5m) 50s ago 36m 50.5M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 5e50731ccbe8 2024-05-06T14:02:53.752 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (6m) 15s ago 34m 44.2M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 61cd48b3d0ff 2024-05-06T14:02:53.752 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (4m) 50s ago 34m 40.9M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 8a473cc2ffd1 2024-05-06T14:02:53.752 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (8m) 50s ago 30m 15.9M - 1.5.0 0da6a335fe13 67f45fa4968b 2024-05-06T14:02:53.752 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (8m) 15s ago 30m 16.1M - 1.5.0 0da6a335fe13 02081b71a5dd 2024-05-06T14:02:53.752 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (3m) 50s ago 34m 43.8M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 fbe876a933ed 2024-05-06T14:02:53.752 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (2m) 50s ago 33m 44.5M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 6f664e567c93 2024-05-06T14:02:53.752 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (107s) 50s ago 33m 43.8M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 a4c3266b87f8 2024-05-06T14:02:53.752 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (55s) 50s ago 33m 12.6M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 7c9e6dbacb88 2024-05-06T14:02:53.752 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (20s) 15s ago 32m 11.7M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 345a6ce2c7c3 2024-05-06T14:02:53.752 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (32m) 15s ago 32m 56.5M 3248M 17.2.0 e1d6a67b021e a55bd4122f6f 2024-05-06T14:02:53.752 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (31m) 15s ago 31m 55.3M 3248M 17.2.0 e1d6a67b021e 362938a0b357 2024-05-06T14:02:53.752 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (31m) 15s ago 31m 59.2M 3248M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T14:02:53.752 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (8m) 15s ago 30m 72.9M - 2.43.0 a07b618ecd1d 10d5c513bb9b 2024-05-06T14:02:53.752 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (29m) 50s ago 29m 79.4M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T14:02:53.752 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (29m) 15s ago 29m 78.7M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T14:02:54.050 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:02:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:02:53] "GET /metrics HTTP/1.1" 200 38056 "" "Prometheus/2.43.0" 2024-05-06T14:02:54.153 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:02:54.153 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-05-06T14:02:54.153 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T14:02:54.153 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:02:54.153 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-05-06T14:02:54.153 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T14:02:54.153 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:02:54.153 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-05-06T14:02:54.153 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-05-06T14:02:54.154 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 5 2024-05-06T14:02:54.154 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:02:54.154 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw": { 2024-05-06T14:02:54.154 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T14:02:54.154 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:02:54.154 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-05-06T14:02:54.154 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-05-06T14:02:54.154 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 10 2024-05-06T14:02:54.154 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T14:02:54.154 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:02:54.470 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:54 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:02:54.470 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:54 smithi170 ceph-mon[65866]: from='client.54504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:02:54.470 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:54 smithi170 ceph-mon[65866]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:02:54.470 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:54 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/1162992583' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:02:54.487 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:02:54.487 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T14:02:54.487 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-05-06T14:02:54.487 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-05-06T14:02:54.488 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [], 2024-05-06T14:02:54.488 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "5/8 daemons upgraded", 2024-05-06T14:02:54.488 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "Currently upgrading osd daemons", 2024-05-06T14:02:54.488 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-05-06T14:02:54.488 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:02:54.488 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:54 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:02:54.488 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:54 smithi077 ceph-mon[89212]: from='client.54504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:02:54.488 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:54 smithi077 ceph-mon[89212]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:02:54.488 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:54 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/1162992583' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:02:54.488 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:54 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:02:54.489 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:54 smithi077 ceph-mon[84247]: from='client.54504 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:02:54.489 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:54 smithi077 ceph-mon[84247]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:02:54.489 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:54 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/1162992583' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:02:55.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:55 smithi170 ceph-mon[65866]: from='client.64444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:02:55.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:55 smithi170 ceph-mon[65866]: from='client.64450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:02:55.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:55 smithi077 ceph-mon[84247]: from='client.64444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:02:55.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:55 smithi077 ceph-mon[84247]: from='client.64450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:02:55.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:55 smithi077 ceph-mon[89212]: from='client.64444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:02:55.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:55 smithi077 ceph-mon[89212]: from='client.64450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:02:56.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:56 smithi077 ceph-mon[84247]: from='client.54528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:02:56.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:56 smithi077 ceph-mon[84247]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:02:56.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:56 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:56.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:56 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:02:56.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:56 smithi077 ceph-mon[89212]: from='client.54528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:02:56.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:56 smithi077 ceph-mon[89212]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:02:56.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:56 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:56.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:56 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:02:56.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:56 smithi170 ceph-mon[65866]: from='client.54528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:02:56.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:56 smithi170 ceph-mon[65866]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:02:56.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:56 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:02:56.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:56 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:02:58.476 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:58 smithi170 ceph-mon[65866]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:02:58.476 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:58 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-06T14:02:58.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:58 smithi077 ceph-mon[84247]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:02:58.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:58 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-06T14:02:58.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:58 smithi077 ceph-mon[89212]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:02:58.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:58 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-06T14:02:59.489 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:59 smithi170 ceph-mon[65866]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-06T14:02:59.489 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:02:59 smithi170 ceph-mon[65866]: Upgrade: osd.5 is safe to restart 2024-05-06T14:02:59.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:59 smithi077 ceph-mon[89212]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-06T14:02:59.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:02:59 smithi077 ceph-mon[89212]: Upgrade: osd.5 is safe to restart 2024-05-06T14:02:59.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:59 smithi077 ceph-mon[84247]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-05-06T14:02:59.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:02:59 smithi077 ceph-mon[84247]: Upgrade: osd.5 is safe to restart 2024-05-06T14:03:00.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:00 smithi170 ceph-mon[65866]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:03:00.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:00 smithi170 ceph-mon[65866]: Upgrade: Updating osd.5 2024-05-06T14:03:00.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:00 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:00.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:00 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-06T14:03:00.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:00 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:03:00.325 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:00 smithi170 ceph-mon[65866]: Deploying daemon osd.5 on smithi170 2024-05-06T14:03:00.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:00 smithi077 ceph-mon[84247]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:03:00.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:00 smithi077 ceph-mon[84247]: Upgrade: Updating osd.5 2024-05-06T14:03:00.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:00 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:00.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:00 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-06T14:03:00.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:00 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:03:00.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:00 smithi077 ceph-mon[84247]: Deploying daemon osd.5 on smithi170 2024-05-06T14:03:00.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:00 smithi077 ceph-mon[89212]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:03:00.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:00 smithi077 ceph-mon[89212]: Upgrade: Updating osd.5 2024-05-06T14:03:00.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:00 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:00.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:00 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-06T14:03:00.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:00 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:03:00.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:00 smithi077 ceph-mon[89212]: Deploying daemon osd.5 on smithi170 2024-05-06T14:03:01.360 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:01 smithi170 ceph-mon[65866]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:03:01.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:01 smithi077 ceph-mon[84247]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:03:01.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:01 smithi077 ceph-mon[89212]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:03:03.035 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:02 smithi170 systemd[1]: Stopping Ceph osd.5 for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:03:03.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:03 smithi170 ceph-mon[65866]: osd.5 marked itself down and dead 2024-05-06T14:03:03.324 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:03 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[29596]: 2024-05-06T14:03:03.090+0000 7f2d98ec2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:03:03.324 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:03 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[29596]: 2024-05-06T14:03:03.090+0000 7f2d98ec2700 -1 osd.5 138 *** Got signal Terminated *** 2024-05-06T14:03:03.324 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:03 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[29596]: 2024-05-06T14:03:03.090+0000 7f2d98ec2700 -1 osd.5 138 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T14:03:03.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:03 smithi077 ceph-mon[84247]: osd.5 marked itself down and dead 2024-05-06T14:03:03.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:03 smithi077 ceph-mon[89212]: osd.5 marked itself down and dead 2024-05-06T14:03:03.706 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:03 smithi170 podman[80156]: 2024-05-06 14:03:03.457579791 +0000 UTC m=+0.502001585 container died a55bd4122f6f0adeee586a7fbaf9e99483a072c36346b94e4677a5b8089ec2d6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5, GIT_BRANCH=HEAD, version=8, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, ceph=True, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., io.buildah.version=1.19.8) 2024-05-06T14:03:03.983 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:03 smithi170 podman[80156]: 2024-05-06 14:03:03.807551166 +0000 UTC m=+0.851972949 container cleanup a55bd4122f6f0adeee586a7fbaf9e99483a072c36346b94e4677a5b8089ec2d6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5, vendor=Red Hat, Inc., RELEASE=HEAD, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, io.openshift.expose-services=, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, distribution-scope=public, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, version=8) 2024-05-06T14:03:03.983 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:03 smithi170 bash[80156]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5 2024-05-06T14:03:04.050 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:03:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:03:03] "GET /metrics HTTP/1.1" 200 38196 "" "Prometheus/2.43.0" 2024-05-06T14:03:04.324 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:04 smithi170 podman[80184]: 2024-05-06 14:03:04.117710387 +0000 UTC m=+0.655648497 container remove a55bd4122f6f0adeee586a7fbaf9e99483a072c36346b94e4677a5b8089ec2d6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, architecture=x86_64, version=8, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, maintainer=Guillaume Abrioux , name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, ceph=True, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container) 2024-05-06T14:03:04.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:04 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:03:04.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:04 smithi170 ceph-mon[65866]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T14:03:04.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:04 smithi170 ceph-mon[65866]: osdmap e139: 8 total, 7 up, 8 in 2024-05-06T14:03:04.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:04 smithi170 ceph-mon[65866]: pgmap v190: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T14:03:04.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:04 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:03:04.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:04 smithi077 ceph-mon[89212]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T14:03:04.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:04 smithi077 ceph-mon[89212]: osdmap e139: 8 total, 7 up, 8 in 2024-05-06T14:03:04.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:04 smithi077 ceph-mon[89212]: pgmap v190: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T14:03:04.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:04 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:03:04.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:04 smithi077 ceph-mon[84247]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T14:03:04.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:04 smithi077 ceph-mon[84247]: osdmap e139: 8 total, 7 up, 8 in 2024-05-06T14:03:04.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:04 smithi077 ceph-mon[84247]: pgmap v190: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T14:03:04.786 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:04 smithi170 podman[80278]: 2024-05-06 14:03:04.443733216 +0000 UTC m=+0.112955469 container create 27ddb6eb99c7a05651f7cc1017906949987b9296edf4d4ca4a14fdc3d611b909 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-06T14:03:04.786 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:04 smithi170 podman[80278]: 2024-05-06 14:03:04.349209676 +0000 UTC m=+0.018431939 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:03:04.786 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:04 smithi170 podman[80278]: 2024-05-06 14:03:04.687968483 +0000 UTC m=+0.357190737 container init 27ddb6eb99c7a05651f7cc1017906949987b9296edf4d4ca4a14fdc3d611b909 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-06T14:03:04.786 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:04 smithi170 podman[80278]: 2024-05-06 14:03:04.691064192 +0000 UTC m=+0.360286455 container start 27ddb6eb99c7a05651f7cc1017906949987b9296edf4d4ca4a14fdc3d611b909 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5-deactivate, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T14:03:05.074 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:04 smithi170 podman[80278]: 2024-05-06 14:03:04.788112443 +0000 UTC m=+0.457334718 container attach 27ddb6eb99c7a05651f7cc1017906949987b9296edf4d4ca4a14fdc3d611b909 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-05-06T14:03:05.074 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:04 smithi170 podman[80278]: 2024-05-06 14:03:04.989863811 +0000 UTC m=+0.659086066 container died 27ddb6eb99c7a05651f7cc1017906949987b9296edf4d4ca4a14fdc3d611b909 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-05-06T14:03:05.444 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:05 smithi170 ceph-mon[65866]: osdmap e140: 8 total, 7 up, 8 in 2024-05-06T14:03:05.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:05 smithi077 ceph-mon[84247]: osdmap e140: 8 total, 7 up, 8 in 2024-05-06T14:03:05.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:05 smithi077 ceph-mon[89212]: osdmap e140: 8 total, 7 up, 8 in 2024-05-06T14:03:05.802 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:05 smithi170 podman[80303]: 2024-05-06 14:03:05.558762731 +0000 UTC m=+0.563815434 container remove 27ddb6eb99c7a05651f7cc1017906949987b9296edf4d4ca4a14fdc3d611b909 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-05-06T14:03:05.803 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:05 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.5.service: Deactivated successfully. 2024-05-06T14:03:05.803 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:05 smithi170 systemd[1]: Stopped Ceph osd.5 for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T14:03:05.803 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:05 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.5.service: Consumed 16.723s CPU time. 2024-05-06T14:03:06.074 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:05 smithi170 systemd[1]: Starting Ceph osd.5 for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:03:06.074 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:06 smithi170 podman[80408]: 2024-05-06 14:03:05.927701388 +0000 UTC m=+0.018496212 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:03:06.074 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:06 smithi170 podman[80408]: 2024-05-06 14:03:06.034832095 +0000 UTC m=+0.125626930 container create 836eb91643cc43a40df5e644e7982d3e252c82bc602288fdc72a3548348dc09d (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5-activate, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-06T14:03:06.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:06 smithi077 ceph-mon[84247]: pgmap v192: 161 pgs: 3 active+undersized+degraded, 23 peering, 10 stale+active+clean, 3 active+undersized, 122 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 9/627 objects degraded (1.435%) 2024-05-06T14:03:06.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:06 smithi077 ceph-mon[84247]: Health check failed: Reduced data availability: 6 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-05-06T14:03:06.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:06 smithi077 ceph-mon[84247]: Health check failed: Degraded data redundancy: 9/627 objects degraded (1.435%), 3 pgs degraded (PG_DEGRADED) 2024-05-06T14:03:06.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:06 smithi077 ceph-mon[89212]: pgmap v192: 161 pgs: 3 active+undersized+degraded, 23 peering, 10 stale+active+clean, 3 active+undersized, 122 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 9/627 objects degraded (1.435%) 2024-05-06T14:03:06.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:06 smithi077 ceph-mon[89212]: Health check failed: Reduced data availability: 6 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-05-06T14:03:06.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:06 smithi077 ceph-mon[89212]: Health check failed: Degraded data redundancy: 9/627 objects degraded (1.435%), 3 pgs degraded (PG_DEGRADED) 2024-05-06T14:03:06.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:06 smithi170 ceph-mon[65866]: pgmap v192: 161 pgs: 3 active+undersized+degraded, 23 peering, 10 stale+active+clean, 3 active+undersized, 122 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 9/627 objects degraded (1.435%) 2024-05-06T14:03:06.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:06 smithi170 ceph-mon[65866]: Health check failed: Reduced data availability: 6 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-05-06T14:03:06.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:06 smithi170 ceph-mon[65866]: Health check failed: Degraded data redundancy: 9/627 objects degraded (1.435%), 3 pgs degraded (PG_DEGRADED) 2024-05-06T14:03:06.574 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:06 smithi170 podman[80408]: 2024-05-06 14:03:06.287827418 +0000 UTC m=+0.378622210 container init 836eb91643cc43a40df5e644e7982d3e252c82bc602288fdc72a3548348dc09d (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5-activate, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T14:03:06.574 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:06 smithi170 podman[80408]: 2024-05-06 14:03:06.291177813 +0000 UTC m=+0.381972609 container start 836eb91643cc43a40df5e644e7982d3e252c82bc602288fdc72a3548348dc09d (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5-activate, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-05-06T14:03:06.574 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:06 smithi170 podman[80408]: 2024-05-06 14:03:06.37111445 +0000 UTC m=+0.461909271 container attach 836eb91643cc43a40df5e644e7982d3e252c82bc602288fdc72a3548348dc09d (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5-activate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-06T14:03:07.074 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5-activate[80423]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-06T14:03:07.074 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5-activate[80423]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-05-06T14:03:07.074 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:06 smithi170 bash[80408]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-06T14:03:07.074 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:06 smithi170 bash[80408]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-05-06T14:03:07.075 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:06 smithi170 bash[80408]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-05-06T14:03:07.075 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:06 smithi170 bash[80408]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-05-06T14:03:07.075 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:06 smithi170 bash[80408]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-05-06T14:03:07.075 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:06 smithi170 bash[80408]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-06T14:03:07.075 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:06 smithi170 bash[80408]: --> ceph-volume raw activate successful for osd ID: 5 2024-05-06T14:03:07.075 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5-activate[80423]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-05-06T14:03:07.075 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5-activate[80423]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-05-06T14:03:07.075 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5-activate[80423]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-05-06T14:03:07.075 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5-activate[80423]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-06T14:03:07.075 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5-activate[80423]: --> ceph-volume raw activate successful for osd ID: 5 2024-05-06T14:03:07.075 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:06 smithi170 podman[80408]: 2024-05-06 14:03:06.751244967 +0000 UTC m=+0.842039761 container died 836eb91643cc43a40df5e644e7982d3e252c82bc602288fdc72a3548348dc09d (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5-activate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-70ce953, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240430) 2024-05-06T14:03:07.757 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:07 smithi170 podman[80555]: 2024-05-06 14:03:07.423361743 +0000 UTC m=+0.667118802 container remove 836eb91643cc43a40df5e644e7982d3e252c82bc602288fdc72a3548348dc09d (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5-activate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-06T14:03:07.757 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:07 smithi170 podman[80619]: 2024-05-06 14:03:07.541536152 +0000 UTC m=+0.018268782 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:03:07.757 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:07 smithi170 podman[80619]: 2024-05-06 14:03:07.651579208 +0000 UTC m=+0.128311816 container create fa8ea71417a67467ae2b02853de95b65cc26a7b63a9c2f0981cf160915a336ca (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-06T14:03:08.173 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:07 smithi170 podman[80619]: 2024-05-06 14:03:07.879213329 +0000 UTC m=+0.355945938 container init fa8ea71417a67467ae2b02853de95b65cc26a7b63a9c2f0981cf160915a336ca (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430) 2024-05-06T14:03:08.173 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:07 smithi170 podman[80619]: 2024-05-06 14:03:07.882096615 +0000 UTC m=+0.358829222 container start fa8ea71417a67467ae2b02853de95b65cc26a7b63a9c2f0981cf160915a336ca (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-05-06T14:03:08.174 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:07 smithi170 ceph-osd[80638]: -- 172.21.15.170:0/1781524761 <== mon.2 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fc12d09680 con 0x55fc12cd2c00 2024-05-06T14:03:08.174 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:07 smithi170 bash[80619]: fa8ea71417a67467ae2b02853de95b65cc26a7b63a9c2f0981cf160915a336ca 2024-05-06T14:03:08.174 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:07 smithi170 systemd[1]: Started Ceph osd.5 for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T14:03:08.429 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:08 smithi170 ceph-mon[65866]: pgmap v193: 161 pgs: 14 active+undersized+degraded, 23 peering, 22 active+undersized, 102 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 42/627 objects degraded (6.699%) 2024-05-06T14:03:08.430 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:08 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:08.430 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:08 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:08.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:08 smithi077 ceph-mon[84247]: pgmap v193: 161 pgs: 14 active+undersized+degraded, 23 peering, 22 active+undersized, 102 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 42/627 objects degraded (6.699%) 2024-05-06T14:03:08.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:08 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:08.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:08 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:08.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:08 smithi077 ceph-mon[89212]: pgmap v193: 161 pgs: 14 active+undersized+degraded, 23 peering, 22 active+undersized, 102 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 42/627 objects degraded (6.699%) 2024-05-06T14:03:08.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:08 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:08.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:08 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:08.815 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:03:08.431+0000 7f34c55517c0 -1 Falling back to public interface 2024-05-06T14:03:09.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:09 smithi077 ceph-mon[84247]: pgmap v194: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-05-06T14:03:09.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:09 smithi077 ceph-mon[84247]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 9 pgs peering) 2024-05-06T14:03:09.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:09 smithi077 ceph-mon[89212]: pgmap v194: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-05-06T14:03:09.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:09 smithi077 ceph-mon[89212]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 9 pgs peering) 2024-05-06T14:03:09.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:09 smithi170 ceph-mon[65866]: pgmap v194: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-05-06T14:03:09.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:09 smithi170 ceph-mon[65866]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 9 pgs peering) 2024-05-06T14:03:11.324 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:11 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:03:11.130+0000 7f34c55517c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-05-06T14:03:11.324 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:11 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:03:11.158+0000 7f34c55517c0 -1 osd.5 138 log_to_monitors true 2024-05-06T14:03:12.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:12 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:12.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:12 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:03:12.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:12 smithi077 ceph-mon[84247]: pgmap v195: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-05-06T14:03:12.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:12 smithi077 ceph-mon[84247]: from='osd.5 [v2:172.21.15.170:6808/3698344853,v1:172.21.15.170:6809/3698344853]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T14:03:12.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:12 smithi077 ceph-mon[84247]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T14:03:12.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:12 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:12.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:12 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:03:12.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:12 smithi077 ceph-mon[89212]: pgmap v195: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-05-06T14:03:12.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:12 smithi077 ceph-mon[89212]: from='osd.5 [v2:172.21.15.170:6808/3698344853,v1:172.21.15.170:6809/3698344853]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T14:03:12.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:12 smithi077 ceph-mon[89212]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T14:03:12.574 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:03:12 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:03:12.177+0000 7f34b69a8700 -1 osd.5 138 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T14:03:12.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:12 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:12.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:12 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:03:12.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:12 smithi170 ceph-mon[65866]: pgmap v195: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-05-06T14:03:12.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:12 smithi170 ceph-mon[65866]: from='osd.5 [v2:172.21.15.170:6808/3698344853,v1:172.21.15.170:6809/3698344853]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T14:03:12.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:12 smithi170 ceph-mon[65866]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-06T14:03:13.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:13 smithi077 ceph-mon[84247]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-06T14:03:13.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:13 smithi077 ceph-mon[84247]: from='osd.5 [v2:172.21.15.170:6808/3698344853,v1:172.21.15.170:6809/3698344853]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T14:03:13.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:13 smithi077 ceph-mon[84247]: osdmap e141: 8 total, 7 up, 8 in 2024-05-06T14:03:13.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:13 smithi077 ceph-mon[84247]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T14:03:13.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:13 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:13.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:13 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:13.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:13 smithi077 ceph-mon[84247]: Health check update: Degraded data redundancy: 71/627 objects degraded (11.324%), 20 pgs degraded (PG_DEGRADED) 2024-05-06T14:03:13.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:13 smithi077 ceph-mon[89212]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-06T14:03:13.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:13 smithi077 ceph-mon[89212]: from='osd.5 [v2:172.21.15.170:6808/3698344853,v1:172.21.15.170:6809/3698344853]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T14:03:13.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:13 smithi077 ceph-mon[89212]: osdmap e141: 8 total, 7 up, 8 in 2024-05-06T14:03:13.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:13 smithi077 ceph-mon[89212]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T14:03:13.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:13 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:13.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:13 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:13.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:13 smithi077 ceph-mon[89212]: Health check update: Degraded data redundancy: 71/627 objects degraded (11.324%), 20 pgs degraded (PG_DEGRADED) 2024-05-06T14:03:13.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:13 smithi170 ceph-mon[65866]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-06T14:03:13.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:13 smithi170 ceph-mon[65866]: from='osd.5 [v2:172.21.15.170:6808/3698344853,v1:172.21.15.170:6809/3698344853]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T14:03:13.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:13 smithi170 ceph-mon[65866]: osdmap e141: 8 total, 7 up, 8 in 2024-05-06T14:03:13.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:13 smithi170 ceph-mon[65866]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T14:03:13.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:13 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:13.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:13 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:13.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:13 smithi170 ceph-mon[65866]: Health check update: Degraded data redundancy: 71/627 objects degraded (11.324%), 20 pgs degraded (PG_DEGRADED) 2024-05-06T14:03:14.050 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:03:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:03:13] "GET /metrics HTTP/1.1" 200 38199 "" "Prometheus/2.43.0" 2024-05-06T14:03:14.292 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:14 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:03:14.293 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:14 smithi170 ceph-mon[65866]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T14:03:14.293 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:14 smithi170 ceph-mon[65866]: pgmap v197: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s; 71/627 objects degraded (11.324%) 2024-05-06T14:03:14.293 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:14 smithi170 ceph-mon[65866]: osd.5 [v2:172.21.15.170:6808/3698344853,v1:172.21.15.170:6809/3698344853] boot 2024-05-06T14:03:14.293 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:14 smithi170 ceph-mon[65866]: osdmap e142: 8 total, 8 up, 8 in 2024-05-06T14:03:14.293 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:14 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T14:03:14.293 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:14 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:14.293 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:14 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:14.293 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:14 smithi170 ceph-mon[65866]: osdmap e143: 8 total, 8 up, 8 in 2024-05-06T14:03:14.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:14 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:03:14.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:14 smithi077 ceph-mon[84247]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T14:03:14.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:14 smithi077 ceph-mon[84247]: pgmap v197: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s; 71/627 objects degraded (11.324%) 2024-05-06T14:03:14.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:14 smithi077 ceph-mon[84247]: osd.5 [v2:172.21.15.170:6808/3698344853,v1:172.21.15.170:6809/3698344853] boot 2024-05-06T14:03:14.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:14 smithi077 ceph-mon[84247]: osdmap e142: 8 total, 8 up, 8 in 2024-05-06T14:03:14.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:14 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T14:03:14.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:14 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:14.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:14 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:14.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:14 smithi077 ceph-mon[84247]: osdmap e143: 8 total, 8 up, 8 in 2024-05-06T14:03:14.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:14 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:03:14.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:14 smithi077 ceph-mon[89212]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T14:03:14.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:14 smithi077 ceph-mon[89212]: pgmap v197: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s; 71/627 objects degraded (11.324%) 2024-05-06T14:03:14.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:14 smithi077 ceph-mon[89212]: osd.5 [v2:172.21.15.170:6808/3698344853,v1:172.21.15.170:6809/3698344853] boot 2024-05-06T14:03:14.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:14 smithi077 ceph-mon[89212]: osdmap e142: 8 total, 8 up, 8 in 2024-05-06T14:03:14.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:14 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-06T14:03:14.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:14 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:14.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:14 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:14.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:14 smithi077 ceph-mon[89212]: osdmap e143: 8 total, 8 up, 8 in 2024-05-06T14:03:16.479 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:16 smithi170 ceph-mon[65866]: pgmap v200: 161 pgs: 13 peering, 14 active+undersized+degraded, 27 active+undersized, 107 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 46/627 objects degraded (7.337%) 2024-05-06T14:03:16.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:16 smithi077 ceph-mon[84247]: pgmap v200: 161 pgs: 13 peering, 14 active+undersized+degraded, 27 active+undersized, 107 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 46/627 objects degraded (7.337%) 2024-05-06T14:03:16.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:16 smithi077 ceph-mon[89212]: pgmap v200: 161 pgs: 13 peering, 14 active+undersized+degraded, 27 active+undersized, 107 active+clean; 585 KiB data, 197 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 46/627 objects degraded (7.337%) 2024-05-06T14:03:17.549 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:03:17 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T14:03:17.068+0000 7f5cd1795700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-05-06T14:03:18.300 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:18.300 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:18.300 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:03:18.300 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:03:18.300 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:18.300 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:03:18.300 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:03:18.301 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:03:18.301 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:03:18.301 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-06T14:03:18.301 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[89212]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-06T14:03:18.301 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[89212]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-05-06T14:03:18.301 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[89212]: pgmap v201: 161 pgs: 13 peering, 7 active+undersized+degraded, 14 active+undersized, 127 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 31/627 objects degraded (4.944%) 2024-05-06T14:03:18.301 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[89212]: Health check update: Degraded data redundancy: 46/627 objects degraded (7.337%), 14 pgs degraded (PG_DEGRADED) 2024-05-06T14:03:18.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:18.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:18.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:03:18.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:03:18.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:03:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:03:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:03:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:03:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-06T14:03:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[84247]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-06T14:03:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[84247]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-05-06T14:03:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[84247]: pgmap v201: 161 pgs: 13 peering, 7 active+undersized+degraded, 14 active+undersized, 127 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 31/627 objects degraded (4.944%) 2024-05-06T14:03:18.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:17 smithi077 ceph-mon[84247]: Health check update: Degraded data redundancy: 46/627 objects degraded (7.337%), 14 pgs degraded (PG_DEGRADED) 2024-05-06T14:03:18.323 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:17 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:18.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:17 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:18.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:17 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:03:18.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:17 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:03:18.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:17 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:18.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:17 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:03:18.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:17 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:03:18.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:17 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:03:18.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:17 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:03:18.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:17 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-06T14:03:18.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:17 smithi170 ceph-mon[65866]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-06T14:03:18.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:17 smithi170 ceph-mon[65866]: Upgrade: unsafe to stop osd(s) at this time (18 PGs are or would become offline) 2024-05-06T14:03:18.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:17 smithi170 ceph-mon[65866]: pgmap v201: 161 pgs: 13 peering, 7 active+undersized+degraded, 14 active+undersized, 127 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 31/627 objects degraded (4.944%) 2024-05-06T14:03:18.324 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:17 smithi170 ceph-mon[65866]: Health check update: Degraded data redundancy: 46/627 objects degraded (7.337%), 14 pgs degraded (PG_DEGRADED) 2024-05-06T14:03:19.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:19 smithi077 ceph-mon[89212]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-05-06T14:03:19.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:19 smithi077 ceph-mon[89212]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/627 objects degraded (4.944%), 7 pgs degraded) 2024-05-06T14:03:19.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:19 smithi077 ceph-mon[89212]: Cluster is now healthy 2024-05-06T14:03:19.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:19 smithi077 ceph-mon[84247]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-05-06T14:03:19.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:19 smithi077 ceph-mon[84247]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/627 objects degraded (4.944%), 7 pgs degraded) 2024-05-06T14:03:19.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:19 smithi077 ceph-mon[84247]: Cluster is now healthy 2024-05-06T14:03:19.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:19 smithi170 ceph-mon[65866]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-05-06T14:03:19.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:19 smithi170 ceph-mon[65866]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/627 objects degraded (4.944%), 7 pgs degraded) 2024-05-06T14:03:19.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:19 smithi170 ceph-mon[65866]: Cluster is now healthy 2024-05-06T14:03:22.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:22 smithi077 ceph-mon[84247]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-05-06T14:03:22.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:22 smithi077 ceph-mon[89212]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-05-06T14:03:22.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:22 smithi170 ceph-mon[65866]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-05-06T14:03:24.050 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:03:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:03:23] "GET /metrics HTTP/1.1" 200 38199 "" "Prometheus/2.43.0" 2024-05-06T14:03:24.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:24 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:03:24.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:24 smithi077 ceph-mon[84247]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T14:03:24.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:24 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:03:24.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:24 smithi077 ceph-mon[89212]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T14:03:24.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:24 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:03:24.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:24 smithi170 ceph-mon[65866]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T14:03:24.851 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T14:03:25.500 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T14:03:25.501 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (8m) 81s ago 30m 26.6M - 0.25.0 c8568f914cd2 997175e3e64e 2024-05-06T14:03:25.501 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (7m) 13s ago 30m 81.0M - 9.4.12 2bacad6d85d8 140bb9a2760a 2024-05-06T14:03:25.501 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (8m) 81s ago 29m 64.2M - 3.5 e1d6a67b021e 9353a303fb55 2024-05-06T14:03:25.501 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283,8765 running (9m) 13s ago 35m 452M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 125521be53a3 2024-05-06T14:03:25.501 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (19m) 81s ago 36m 509M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T14:03:25.501 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (5m) 81s ago 36m 50.5M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 5e50731ccbe8 2024-05-06T14:03:25.501 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (6m) 13s ago 35m 45.4M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 61cd48b3d0ff 2024-05-06T14:03:25.501 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (5m) 81s ago 35m 40.9M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 8a473cc2ffd1 2024-05-06T14:03:25.501 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (9m) 81s ago 31m 15.9M - 1.5.0 0da6a335fe13 67f45fa4968b 2024-05-06T14:03:25.501 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (9m) 13s ago 31m 16.5M - 1.5.0 0da6a335fe13 02081b71a5dd 2024-05-06T14:03:25.501 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (3m) 81s ago 34m 43.8M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 fbe876a933ed 2024-05-06T14:03:25.501 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (3m) 81s ago 34m 44.5M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 6f664e567c93 2024-05-06T14:03:25.501 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (2m) 81s ago 34m 43.8M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 a4c3266b87f8 2024-05-06T14:03:25.501 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (87s) 81s ago 33m 12.6M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 7c9e6dbacb88 2024-05-06T14:03:25.502 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (52s) 13s ago 33m 43.2M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 345a6ce2c7c3 2024-05-06T14:03:25.502 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (17s) 13s ago 32m 11.7M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 fa8ea71417a6 2024-05-06T14:03:25.502 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (32m) 13s ago 32m 56.0M 3248M 17.2.0 e1d6a67b021e 362938a0b357 2024-05-06T14:03:25.502 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (32m) 13s ago 32m 59.7M 3248M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T14:03:25.502 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (8m) 13s ago 31m 72.9M - 2.43.0 a07b618ecd1d 10d5c513bb9b 2024-05-06T14:03:25.502 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (29m) 81s ago 29m 79.4M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T14:03:25.502 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (29m) 13s ago 29m 78.9M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T14:03:25.892 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:03:25.892 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-05-06T14:03:25.892 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T14:03:25.892 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:03:25.892 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-05-06T14:03:25.892 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T14:03:25.892 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:03:25.892 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-05-06T14:03:25.892 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-05-06T14:03:25.892 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 6 2024-05-06T14:03:25.892 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:03:25.893 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw": { 2024-05-06T14:03:25.893 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T14:03:25.893 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:03:25.893 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-05-06T14:03:25.893 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-05-06T14:03:25.893 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 11 2024-05-06T14:03:25.893 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T14:03:25.893 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:03:26.229 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:03:26.229 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T14:03:26.229 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-05-06T14:03:26.229 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-05-06T14:03:26.229 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [], 2024-05-06T14:03:26.229 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "6/8 daemons upgraded", 2024-05-06T14:03:26.229 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "Currently upgrading osd daemons", 2024-05-06T14:03:26.229 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-05-06T14:03:26.229 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:03:26.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:26 smithi077 ceph-mon[84247]: from='client.64474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:03:26.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:26 smithi077 ceph-mon[84247]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T14:03:26.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:26 smithi077 ceph-mon[84247]: from='client.64480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:03:26.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:26 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/2326914581' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:03:26.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:26 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:26.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:26 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:03:26.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:26 smithi077 ceph-mon[89212]: from='client.64474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:03:26.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:26 smithi077 ceph-mon[89212]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T14:03:26.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:26 smithi077 ceph-mon[89212]: from='client.64480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:03:26.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:26 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/2326914581' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:03:26.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:26 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:26.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:26 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:03:26.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:26 smithi170 ceph-mon[65866]: from='client.64474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:03:26.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:26 smithi170 ceph-mon[65866]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T14:03:26.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:26 smithi170 ceph-mon[65866]: from='client.64480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:03:26.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:26 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/2326914581' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:03:26.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:26 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:26.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:26 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:03:27.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:27 smithi077 ceph-mon[84247]: from='client.64486 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:03:27.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:27 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:27.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:27 smithi077 ceph-mon[89212]: from='client.64486 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:03:27.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:27 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:27.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:27 smithi170 ceph-mon[65866]: from='client.64486 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:03:27.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:27 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:28.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:28 smithi077 ceph-mon[89212]: from='client.64498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:03:28.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:28 smithi077 ceph-mon[89212]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T14:03:28.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:28 smithi077 ceph-mon[84247]: from='client.64498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:03:28.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:28 smithi077 ceph-mon[84247]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T14:03:28.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:28 smithi170 ceph-mon[65866]: from='client.64498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:03:28.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:28 smithi170 ceph-mon[65866]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T14:03:30.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:30 smithi077 ceph-mon[84247]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-05-06T14:03:30.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:30 smithi077 ceph-mon[89212]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-05-06T14:03:30.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:30 smithi170 ceph-mon[65866]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-05-06T14:03:32.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:32 smithi077 ceph-mon[84247]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:03:32.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-06T14:03:32.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:32 smithi077 ceph-mon[89212]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:03:32.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-06T14:03:32.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:32 smithi170 ceph-mon[65866]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:03:32.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-06T14:03:33.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:33 smithi077 ceph-mon[84247]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-06T14:03:33.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:33 smithi077 ceph-mon[84247]: Upgrade: osd.6 is safe to restart 2024-05-06T14:03:33.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:33 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:03:33.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:33 smithi077 ceph-mon[84247]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:03:33.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:33 smithi077 ceph-mon[89212]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-06T14:03:33.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:33 smithi077 ceph-mon[89212]: Upgrade: osd.6 is safe to restart 2024-05-06T14:03:33.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:33 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:03:33.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:33 smithi077 ceph-mon[89212]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:03:33.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:33 smithi170 ceph-mon[65866]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-05-06T14:03:33.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:33 smithi170 ceph-mon[65866]: Upgrade: osd.6 is safe to restart 2024-05-06T14:03:33.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:33 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:03:33.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:33 smithi170 ceph-mon[65866]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:03:34.049 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:03:33 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:03:33] "GET /metrics HTTP/1.1" 200 38204 "" "Prometheus/2.43.0" 2024-05-06T14:03:35.210 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:34 smithi170 ceph-mon[65866]: Upgrade: Updating osd.6 2024-05-06T14:03:35.210 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:34 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:35.210 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:34 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-06T14:03:35.210 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:34 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:03:35.210 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:34 smithi170 ceph-mon[65866]: Deploying daemon osd.6 on smithi170 2024-05-06T14:03:35.299 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:34 smithi077 ceph-mon[84247]: Upgrade: Updating osd.6 2024-05-06T14:03:35.299 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:34 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:35.299 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:34 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-06T14:03:35.300 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:34 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:03:35.300 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:34 smithi077 ceph-mon[84247]: Deploying daemon osd.6 on smithi170 2024-05-06T14:03:35.300 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:34 smithi077 ceph-mon[89212]: Upgrade: Updating osd.6 2024-05-06T14:03:35.300 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:34 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:35.300 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:34 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-06T14:03:35.300 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:34 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:03:35.300 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:34 smithi077 ceph-mon[89212]: Deploying daemon osd.6 on smithi170 2024-05-06T14:03:36.073 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:35 smithi170 ceph-mon[65866]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:03:36.299 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:35 smithi077 ceph-mon[84247]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:03:36.299 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:35 smithi077 ceph-mon[89212]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:03:37.423 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:37 smithi170 systemd[1]: Stopping Ceph osd.6 for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:03:37.823 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:37 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[32498]: 2024-05-06T14:03:37.424+0000 7f9996343700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:03:37.823 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:37 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[32498]: 2024-05-06T14:03:37.424+0000 7f9996343700 -1 osd.6 143 *** Got signal Terminated *** 2024-05-06T14:03:37.823 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:37 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[32498]: 2024-05-06T14:03:37.424+0000 7f9996343700 -1 osd.6 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T14:03:38.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:38 smithi077 ceph-mon[84247]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-05-06T14:03:38.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:38 smithi077 ceph-mon[84247]: osd.6 marked itself down and dead 2024-05-06T14:03:38.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:38 smithi077 ceph-mon[89212]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-05-06T14:03:38.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:38 smithi077 ceph-mon[89212]: osd.6 marked itself down and dead 2024-05-06T14:03:38.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:38 smithi170 ceph-mon[65866]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-05-06T14:03:38.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:38 smithi170 ceph-mon[65866]: osd.6 marked itself down and dead 2024-05-06T14:03:38.931 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:38 smithi170 podman[85023]: 2024-05-06 14:03:38.662890055 +0000 UTC m=+1.378661105 container died 362938a0b357e2e08d7049cf73612f111ad7789616dd2a99832a89eeb624969e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, release=754, architecture=x86_64, RELEASE=HEAD, GIT_CLEAN=True, name=centos-stream, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, version=8, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-05-06T14:03:39.241 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:39 smithi170 podman[85023]: 2024-05-06 14:03:39.024414901 +0000 UTC m=+1.740185948 container cleanup 362938a0b357e2e08d7049cf73612f111ad7789616dd2a99832a89eeb624969e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, release=754, io.buildah.version=1.19.8, vcs-type=git, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public) 2024-05-06T14:03:39.241 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:39 smithi170 bash[85023]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6 2024-05-06T14:03:39.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:39 smithi077 ceph-mon[84247]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T14:03:39.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:39 smithi077 ceph-mon[84247]: osdmap e144: 8 total, 7 up, 8 in 2024-05-06T14:03:39.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:39 smithi077 ceph-mon[84247]: pgmap v213: 161 pgs: 14 peering, 12 stale+active+clean, 135 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-05-06T14:03:39.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:39 smithi077 ceph-mon[84247]: osdmap e145: 8 total, 7 up, 8 in 2024-05-06T14:03:39.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:39 smithi077 ceph-mon[89212]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T14:03:39.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:39 smithi077 ceph-mon[89212]: osdmap e144: 8 total, 7 up, 8 in 2024-05-06T14:03:39.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:39 smithi077 ceph-mon[89212]: pgmap v213: 161 pgs: 14 peering, 12 stale+active+clean, 135 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-05-06T14:03:39.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:39 smithi077 ceph-mon[89212]: osdmap e145: 8 total, 7 up, 8 in 2024-05-06T14:03:39.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:39 smithi170 ceph-mon[65866]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T14:03:39.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:39 smithi170 ceph-mon[65866]: osdmap e144: 8 total, 7 up, 8 in 2024-05-06T14:03:39.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:39 smithi170 ceph-mon[65866]: pgmap v213: 161 pgs: 14 peering, 12 stale+active+clean, 135 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-05-06T14:03:39.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:39 smithi170 ceph-mon[65866]: osdmap e145: 8 total, 7 up, 8 in 2024-05-06T14:03:39.574 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:39 smithi170 podman[85040]: 2024-05-06 14:03:39.32618198 +0000 UTC m=+0.660875871 container remove 362938a0b357e2e08d7049cf73612f111ad7789616dd2a99832a89eeb624969e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, name=centos-stream, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , version=8, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, com.redhat.component=centos-stream-container, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=) 2024-05-06T14:03:39.911 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:39 smithi170 podman[85131]: 2024-05-06 14:03:39.54863614 +0000 UTC m=+0.018461705 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:03:39.911 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:39 smithi170 podman[85131]: 2024-05-06 14:03:39.65249592 +0000 UTC m=+0.122321453 container create e802941e9b2e42df406eca4669f83df4fc05430fc01b85a02fe8071ab1bef8a3 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240430, RELEASE=squid-70ce953, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-06T14:03:40.200 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:39 smithi170 podman[85131]: 2024-05-06 14:03:39.913985908 +0000 UTC m=+0.383811442 container init e802941e9b2e42df406eca4669f83df4fc05430fc01b85a02fe8071ab1bef8a3 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0) 2024-05-06T14:03:40.200 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:39 smithi170 podman[85131]: 2024-05-06 14:03:39.918127564 +0000 UTC m=+0.387953104 container start e802941e9b2e42df406eca4669f83df4fc05430fc01b85a02fe8071ab1bef8a3 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6-deactivate, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240430) 2024-05-06T14:03:40.200 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:39 smithi170 podman[85131]: 2024-05-06 14:03:39.988898761 +0000 UTC m=+0.458724303 container attach e802941e9b2e42df406eca4669f83df4fc05430fc01b85a02fe8071ab1bef8a3 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, RELEASE=squid-70ce953, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-06T14:03:40.490 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:40 smithi170 ceph-mon[65866]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-05-06T14:03:40.490 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:40 smithi170 podman[85131]: 2024-05-06 14:03:40.203577188 +0000 UTC m=+0.673402740 container died e802941e9b2e42df406eca4669f83df4fc05430fc01b85a02fe8071ab1bef8a3 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-05-06T14:03:40.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:40 smithi077 ceph-mon[84247]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-05-06T14:03:40.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:40 smithi077 ceph-mon[89212]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-05-06T14:03:41.073 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:40 smithi170 podman[85156]: 2024-05-06 14:03:40.800853965 +0000 UTC m=+0.592260886 container remove e802941e9b2e42df406eca4669f83df4fc05430fc01b85a02fe8071ab1bef8a3 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-05-06T14:03:41.073 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:40 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.6.service: Deactivated successfully. 2024-05-06T14:03:41.074 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:40 smithi170 systemd[1]: Stopped Ceph osd.6 for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T14:03:41.074 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:40 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.6.service: Consumed 14.835s CPU time. 2024-05-06T14:03:41.074 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:41 smithi170 systemd[1]: Starting Ceph osd.6 for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:03:41.493 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:41 smithi170 podman[85263]: 2024-05-06 14:03:41.137967387 +0000 UTC m=+0.018980417 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:03:41.493 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:41 smithi170 podman[85263]: 2024-05-06 14:03:41.260625271 +0000 UTC m=+0.141638278 container create 13f137b71a8f7e3c12e94733a258bf3aee3a6cc5a87e6d18be04570d578a2f2b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6-activate, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True) 2024-05-06T14:03:41.493 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:41 smithi170 podman[85263]: 2024-05-06 14:03:41.496532666 +0000 UTC m=+0.377545672 container init 13f137b71a8f7e3c12e94733a258bf3aee3a6cc5a87e6d18be04570d578a2f2b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6-activate, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-05-06T14:03:41.823 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:41 smithi170 podman[85263]: 2024-05-06 14:03:41.499955587 +0000 UTC m=+0.380968594 container start 13f137b71a8f7e3c12e94733a258bf3aee3a6cc5a87e6d18be04570d578a2f2b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6-activate, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-06T14:03:41.823 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:41 smithi170 podman[85263]: 2024-05-06 14:03:41.571346055 +0000 UTC m=+0.452359070 container attach 13f137b71a8f7e3c12e94733a258bf3aee3a6cc5a87e6d18be04570d578a2f2b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6-activate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-70ce953, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True) 2024-05-06T14:03:42.280 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:42 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:42.280 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:42 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:03:42.280 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:42 smithi170 ceph-mon[65866]: pgmap v215: 161 pgs: 17 peering, 10 stale+active+clean, 134 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 51 op/s 2024-05-06T14:03:42.281 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:41 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6-activate[85278]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-06T14:03:42.281 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:41 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6-activate[85278]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-05-06T14:03:42.281 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:41 smithi170 bash[85263]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-06T14:03:42.281 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:41 smithi170 bash[85263]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-05-06T14:03:42.281 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:41 smithi170 bash[85263]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-05-06T14:03:42.281 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:41 smithi170 bash[85263]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-05-06T14:03:42.281 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:41 smithi170 bash[85263]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-05-06T14:03:42.281 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:41 smithi170 bash[85263]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-06T14:03:42.281 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:41 smithi170 bash[85263]: --> ceph-volume raw activate successful for osd ID: 6 2024-05-06T14:03:42.282 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:41 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6-activate[85278]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-05-06T14:03:42.282 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:41 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6-activate[85278]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-05-06T14:03:42.282 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:41 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6-activate[85278]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-05-06T14:03:42.282 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:41 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6-activate[85278]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-06T14:03:42.282 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:41 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6-activate[85278]: --> ceph-volume raw activate successful for osd ID: 6 2024-05-06T14:03:42.282 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:41 smithi170 podman[85263]: 2024-05-06 14:03:41.965980911 +0000 UTC m=+0.846993934 container died 13f137b71a8f7e3c12e94733a258bf3aee3a6cc5a87e6d18be04570d578a2f2b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6-activate, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2) 2024-05-06T14:03:42.413 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:42 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:42.413 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:42 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:03:42.413 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:42 smithi077 ceph-mon[84247]: pgmap v215: 161 pgs: 17 peering, 10 stale+active+clean, 134 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 51 op/s 2024-05-06T14:03:42.413 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:42 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:42.413 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:42 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:03:42.413 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:42 smithi077 ceph-mon[89212]: pgmap v215: 161 pgs: 17 peering, 10 stale+active+clean, 134 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 51 op/s 2024-05-06T14:03:42.823 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:42 smithi170 podman[85405]: 2024-05-06 14:03:42.567052898 +0000 UTC m=+0.591767569 container remove 13f137b71a8f7e3c12e94733a258bf3aee3a6cc5a87e6d18be04570d578a2f2b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6-activate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-06T14:03:42.823 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:42 smithi170 podman[85472]: 2024-05-06 14:03:42.686192573 +0000 UTC m=+0.018240222 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:03:42.823 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:42 smithi170 podman[85472]: 2024-05-06 14:03:42.826805153 +0000 UTC m=+0.158852832 container create 1fbae25547b949cb42e441ed7b34b1818938d31d511d3de23e03dfb4af83bfd4 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-05-06T14:03:43.101 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:43 smithi170 podman[85472]: 2024-05-06 14:03:43.012450005 +0000 UTC m=+0.344497642 container init 1fbae25547b949cb42e441ed7b34b1818938d31d511d3de23e03dfb4af83bfd4 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-06T14:03:43.101 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:43 smithi170 podman[85472]: 2024-05-06 14:03:43.015388457 +0000 UTC m=+0.347436106 container start 1fbae25547b949cb42e441ed7b34b1818938d31d511d3de23e03dfb4af83bfd4 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, RELEASE=squid-70ce953) 2024-05-06T14:03:43.101 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:43 smithi170 ceph-osd[85491]: -- 172.21.15.170:0/1489654363 <== mon.2 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563a6fe0b680 con 0x563a6fdd5400 2024-05-06T14:03:43.559 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:43 smithi170 bash[85472]: 1fbae25547b949cb42e441ed7b34b1818938d31d511d3de23e03dfb4af83bfd4 2024-05-06T14:03:43.559 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:43 smithi170 systemd[1]: Started Ceph osd.6 for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T14:03:43.823 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:43 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:03:43.561+0000 7f090bd747c0 -1 Falling back to public interface 2024-05-06T14:03:44.049 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:03:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:03:43] "GET /metrics HTTP/1.1" 200 38279 "" "Prometheus/2.43.0" 2024-05-06T14:03:44.271 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:44 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:03:44.271 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:44 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:44.271 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:44 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:44.271 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:44 smithi170 ceph-mon[65866]: pgmap v216: 161 pgs: 12 active+undersized+degraded, 17 peering, 1 stale+active+clean, 22 active+undersized, 109 active+clean; 585 KiB data, 202 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 135 op/s; 51/627 objects degraded (8.134%) 2024-05-06T14:03:44.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:44 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:03:44.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:44 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:44.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:44 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:44.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:44 smithi077 ceph-mon[84247]: pgmap v216: 161 pgs: 12 active+undersized+degraded, 17 peering, 1 stale+active+clean, 22 active+undersized, 109 active+clean; 585 KiB data, 202 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 135 op/s; 51/627 objects degraded (8.134%) 2024-05-06T14:03:44.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:44 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:03:44.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:44 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:44.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:44 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:44.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:44 smithi077 ceph-mon[89212]: pgmap v216: 161 pgs: 12 active+undersized+degraded, 17 peering, 1 stale+active+clean, 22 active+undersized, 109 active+clean; 585 KiB data, 202 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 135 op/s; 51/627 objects degraded (8.134%) 2024-05-06T14:03:45.323 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:45 smithi170 ceph-mon[65866]: Health check failed: Degraded data redundancy: 51/627 objects degraded (8.134%), 12 pgs degraded (PG_DEGRADED) 2024-05-06T14:03:45.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:45 smithi077 ceph-mon[84247]: Health check failed: Degraded data redundancy: 51/627 objects degraded (8.134%), 12 pgs degraded (PG_DEGRADED) 2024-05-06T14:03:45.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:45 smithi077 ceph-mon[89212]: Health check failed: Degraded data redundancy: 51/627 objects degraded (8.134%), 12 pgs degraded (PG_DEGRADED) 2024-05-06T14:03:46.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:46 smithi077 ceph-mon[84247]: pgmap v217: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 104 KiB/s rd, 0 B/s wr, 173 op/s; 75/627 objects degraded (11.962%) 2024-05-06T14:03:46.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:46 smithi077 ceph-mon[89212]: pgmap v217: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 104 KiB/s rd, 0 B/s wr, 173 op/s; 75/627 objects degraded (11.962%) 2024-05-06T14:03:46.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:46 smithi170 ceph-mon[65866]: pgmap v217: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 104 KiB/s rd, 0 B/s wr, 173 op/s; 75/627 objects degraded (11.962%) 2024-05-06T14:03:46.573 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:46 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:03:46.286+0000 7f090bd747c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-05-06T14:03:46.573 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:46 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:03:46.319+0000 7f090bd747c0 -1 osd.6 143 log_to_monitors true 2024-05-06T14:03:47.189 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:47 smithi170 ceph-mon[65866]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-05-06T14:03:47.190 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:47 smithi170 ceph-mon[65866]: from='osd.6 [v2:172.21.15.170:6816/302677463,v1:172.21.15.170:6817/302677463]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-06T14:03:47.190 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:47 smithi170 ceph-mon[65866]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-06T14:03:47.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:47 smithi077 ceph-mon[84247]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-05-06T14:03:47.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:47 smithi077 ceph-mon[84247]: from='osd.6 [v2:172.21.15.170:6816/302677463,v1:172.21.15.170:6817/302677463]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-06T14:03:47.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:47 smithi077 ceph-mon[84247]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-06T14:03:47.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:47 smithi077 ceph-mon[89212]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-05-06T14:03:47.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:47 smithi077 ceph-mon[89212]: from='osd.6 [v2:172.21.15.170:6816/302677463,v1:172.21.15.170:6817/302677463]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-06T14:03:47.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:47 smithi077 ceph-mon[89212]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-06T14:03:47.573 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:03:47 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:03:47.192+0000 7f08fd1cb700 -1 osd.6 143 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T14:03:48.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:48 smithi077 ceph-mon[84247]: pgmap v218: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 0 B/s wr, 152 op/s; 75/627 objects degraded (11.962%) 2024-05-06T14:03:48.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:48 smithi077 ceph-mon[84247]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-06T14:03:48.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:48 smithi077 ceph-mon[84247]: from='osd.6 [v2:172.21.15.170:6816/302677463,v1:172.21.15.170:6817/302677463]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T14:03:48.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:48 smithi077 ceph-mon[84247]: osdmap e146: 8 total, 7 up, 8 in 2024-05-06T14:03:48.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:48 smithi077 ceph-mon[84247]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T14:03:48.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:48 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:48.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:48 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:48.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:48 smithi077 ceph-mon[89212]: pgmap v218: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 0 B/s wr, 152 op/s; 75/627 objects degraded (11.962%) 2024-05-06T14:03:48.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:48 smithi077 ceph-mon[89212]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-06T14:03:48.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:48 smithi077 ceph-mon[89212]: from='osd.6 [v2:172.21.15.170:6816/302677463,v1:172.21.15.170:6817/302677463]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T14:03:48.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:48 smithi077 ceph-mon[89212]: osdmap e146: 8 total, 7 up, 8 in 2024-05-06T14:03:48.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:48 smithi077 ceph-mon[89212]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T14:03:48.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:48 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:48.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:48 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:48.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:48 smithi170 ceph-mon[65866]: pgmap v218: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 0 B/s wr, 152 op/s; 75/627 objects degraded (11.962%) 2024-05-06T14:03:48.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:48 smithi170 ceph-mon[65866]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-06T14:03:48.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:48 smithi170 ceph-mon[65866]: from='osd.6 [v2:172.21.15.170:6816/302677463,v1:172.21.15.170:6817/302677463]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T14:03:48.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:48 smithi170 ceph-mon[65866]: osdmap e146: 8 total, 7 up, 8 in 2024-05-06T14:03:48.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:48 smithi170 ceph-mon[65866]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T14:03:48.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:48 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:48.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:48 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:49.506 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:49 smithi170 ceph-mon[65866]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T14:03:49.506 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:49 smithi170 ceph-mon[65866]: osd.6 [v2:172.21.15.170:6816/302677463,v1:172.21.15.170:6817/302677463] boot 2024-05-06T14:03:49.506 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:49 smithi170 ceph-mon[65866]: osdmap e147: 8 total, 8 up, 8 in 2024-05-06T14:03:49.506 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:49 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T14:03:49.506 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:49 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:49.506 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:49 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:49.506 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:49 smithi170 ceph-mon[65866]: osdmap e148: 8 total, 8 up, 8 in 2024-05-06T14:03:49.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:49 smithi077 ceph-mon[84247]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T14:03:49.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:49 smithi077 ceph-mon[84247]: osd.6 [v2:172.21.15.170:6816/302677463,v1:172.21.15.170:6817/302677463] boot 2024-05-06T14:03:49.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:49 smithi077 ceph-mon[84247]: osdmap e147: 8 total, 8 up, 8 in 2024-05-06T14:03:49.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:49 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T14:03:49.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:49 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:49.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:49 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:49.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:49 smithi077 ceph-mon[84247]: osdmap e148: 8 total, 8 up, 8 in 2024-05-06T14:03:49.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:49 smithi077 ceph-mon[89212]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T14:03:49.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:49 smithi077 ceph-mon[89212]: osd.6 [v2:172.21.15.170:6816/302677463,v1:172.21.15.170:6817/302677463] boot 2024-05-06T14:03:49.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:49 smithi077 ceph-mon[89212]: osdmap e147: 8 total, 8 up, 8 in 2024-05-06T14:03:49.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:49 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-06T14:03:49.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:49 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:49.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:49 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:49.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:49 smithi077 ceph-mon[89212]: osdmap e148: 8 total, 8 up, 8 in 2024-05-06T14:03:50.537 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:50 smithi170 ceph-mon[65866]: pgmap v221: 161 pgs: 14 peering, 13 active+undersized+degraded, 26 active+undersized, 108 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 127 op/s; 53/627 objects degraded (8.453%) 2024-05-06T14:03:50.537 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:50 smithi170 ceph-mon[65866]: Health check update: Degraded data redundancy: 53/627 objects degraded (8.453%), 13 pgs degraded (PG_DEGRADED) 2024-05-06T14:03:50.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:50 smithi077 ceph-mon[84247]: pgmap v221: 161 pgs: 14 peering, 13 active+undersized+degraded, 26 active+undersized, 108 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 127 op/s; 53/627 objects degraded (8.453%) 2024-05-06T14:03:50.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:50 smithi077 ceph-mon[84247]: Health check update: Degraded data redundancy: 53/627 objects degraded (8.453%), 13 pgs degraded (PG_DEGRADED) 2024-05-06T14:03:50.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:50 smithi077 ceph-mon[89212]: pgmap v221: 161 pgs: 14 peering, 13 active+undersized+degraded, 26 active+undersized, 108 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 127 op/s; 53/627 objects degraded (8.453%) 2024-05-06T14:03:50.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:50 smithi077 ceph-mon[89212]: Health check update: Degraded data redundancy: 53/627 objects degraded (8.453%), 13 pgs degraded (PG_DEGRADED) 2024-05-06T14:03:52.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:52 smithi077 ceph-mon[84247]: pgmap v223: 161 pgs: 14 peering, 13 active+undersized+degraded, 25 active+undersized, 109 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 53/627 objects degraded (8.453%) 2024-05-06T14:03:52.549 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:03:52 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T14:03:52.373+0000 7f5cd1795700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-05-06T14:03:52.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:52 smithi077 ceph-mon[89212]: pgmap v223: 161 pgs: 14 peering, 13 active+undersized+degraded, 25 active+undersized, 109 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 53/627 objects degraded (8.453%) 2024-05-06T14:03:52.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:52 smithi170 ceph-mon[65866]: pgmap v223: 161 pgs: 14 peering, 13 active+undersized+degraded, 25 active+undersized, 109 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 53/627 objects degraded (8.453%) 2024-05-06T14:03:53.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:53 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:53.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:53 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:53.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:53 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:03:53.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:53 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:03:53.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:53 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:53.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:53 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:03:53.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:53 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:03:53.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:53 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:03:53.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:53 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:03:53.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:53 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-06T14:03:53.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:53 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:53.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:53 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:53.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:53 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:03:53.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:53 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:03:53.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:53 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:53.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:53 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:03:53.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:53 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:03:53.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:53 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:03:53.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:53 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:03:53.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:53 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-06T14:03:53.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:53 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:53.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:53 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:53.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:53 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:03:53.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:53 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:03:53.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:53 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:53.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:53 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:03:53.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:53 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:03:53.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:53 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:03:53.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:53 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:03:53.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:53 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-06T14:03:54.049 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:03:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:03:53] "GET /metrics HTTP/1.1" 200 38279 "" "Prometheus/2.43.0" 2024-05-06T14:03:54.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:54 smithi077 ceph-mon[89212]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-06T14:03:54.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:54 smithi077 ceph-mon[89212]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-05-06T14:03:54.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:54 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:03:54.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:54 smithi077 ceph-mon[89212]: pgmap v224: 161 pgs: 14 peering, 147 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 0 B/s, 6 objects/s recovering 2024-05-06T14:03:54.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:54 smithi077 ceph-mon[89212]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 53/627 objects degraded (8.453%), 13 pgs degraded) 2024-05-06T14:03:54.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:54 smithi077 ceph-mon[89212]: Cluster is now healthy 2024-05-06T14:03:54.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:54 smithi077 ceph-mon[84247]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-06T14:03:54.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:54 smithi077 ceph-mon[84247]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-05-06T14:03:54.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:54 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:03:54.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:54 smithi077 ceph-mon[84247]: pgmap v224: 161 pgs: 14 peering, 147 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 0 B/s, 6 objects/s recovering 2024-05-06T14:03:54.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:54 smithi077 ceph-mon[84247]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 53/627 objects degraded (8.453%), 13 pgs degraded) 2024-05-06T14:03:54.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:54 smithi077 ceph-mon[84247]: Cluster is now healthy 2024-05-06T14:03:54.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:54 smithi170 ceph-mon[65866]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-06T14:03:54.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:54 smithi170 ceph-mon[65866]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-05-06T14:03:54.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:54 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:03:54.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:54 smithi170 ceph-mon[65866]: pgmap v224: 161 pgs: 14 peering, 147 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 0 B/s, 6 objects/s recovering 2024-05-06T14:03:54.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:54 smithi170 ceph-mon[65866]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 53/627 objects degraded (8.453%), 13 pgs degraded) 2024-05-06T14:03:54.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:54 smithi170 ceph-mon[65866]: Cluster is now healthy 2024-05-06T14:03:56.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:56 smithi077 ceph-mon[84247]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s; 0 B/s, 6 objects/s recovering 2024-05-06T14:03:56.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:56 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:56.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:56 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:03:56.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:56 smithi077 ceph-mon[89212]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s; 0 B/s, 6 objects/s recovering 2024-05-06T14:03:56.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:56 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:56.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:56 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:03:56.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:56 smithi170 ceph-mon[65866]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s; 0 B/s, 6 objects/s recovering 2024-05-06T14:03:56.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:56 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:03:56.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:56 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:03:56.583 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T14:03:57.234 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T14:03:57.235 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (8m) 113s ago 31m 26.6M - 0.25.0 c8568f914cd2 997175e3e64e 2024-05-06T14:03:57.235 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (8m) 9s ago 30m 81.5M - 9.4.12 2bacad6d85d8 140bb9a2760a 2024-05-06T14:03:57.235 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (9m) 113s ago 30m 64.2M - 3.5 e1d6a67b021e 9353a303fb55 2024-05-06T14:03:57.235 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283,8765 running (10m) 9s ago 35m 452M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 125521be53a3 2024-05-06T14:03:57.235 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (19m) 113s ago 37m 509M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T14:03:57.235 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (6m) 113s ago 37m 50.5M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 5e50731ccbe8 2024-05-06T14:03:57.235 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (7m) 9s ago 35m 46.9M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 61cd48b3d0ff 2024-05-06T14:03:57.235 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (6m) 113s ago 36m 40.9M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 8a473cc2ffd1 2024-05-06T14:03:57.235 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (9m) 113s ago 31m 15.9M - 1.5.0 0da6a335fe13 67f45fa4968b 2024-05-06T14:03:57.236 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (9m) 9s ago 31m 16.4M - 1.5.0 0da6a335fe13 02081b71a5dd 2024-05-06T14:03:57.236 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (4m) 113s ago 35m 43.8M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 fbe876a933ed 2024-05-06T14:03:57.236 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (3m) 113s ago 34m 44.5M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 6f664e567c93 2024-05-06T14:03:57.236 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (2m) 113s ago 34m 43.8M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 a4c3266b87f8 2024-05-06T14:03:57.236 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (118s) 113s ago 34m 12.6M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 7c9e6dbacb88 2024-05-06T14:03:57.236 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (83s) 9s ago 33m 45.8M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 345a6ce2c7c3 2024-05-06T14:03:57.236 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (49s) 9s ago 33m 43.9M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 fa8ea71417a6 2024-05-06T14:03:57.236 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (14s) 9s ago 33m 15.0M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 1fbae25547b9 2024-05-06T14:03:57.236 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (32m) 9s ago 32m 60.8M 3248M 17.2.0 e1d6a67b021e 8336b964be29 2024-05-06T14:03:57.236 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (9m) 9s ago 31m 73.0M - 2.43.0 a07b618ecd1d 10d5c513bb9b 2024-05-06T14:03:57.236 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (30m) 113s ago 30m 79.4M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T14:03:57.236 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (30m) 9s ago 30m 79.4M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T14:03:57.635 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:03:57.635 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-05-06T14:03:57.635 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T14:03:57.635 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:03:57.636 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-05-06T14:03:57.636 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T14:03:57.636 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:03:57.636 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-05-06T14:03:57.636 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-05-06T14:03:57.636 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 7 2024-05-06T14:03:57.636 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:03:57.636 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw": { 2024-05-06T14:03:57.636 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T14:03:57.636 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:03:57.636 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-05-06T14:03:57.636 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-05-06T14:03:57.636 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 12 2024-05-06T14:03:57.637 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T14:03:57.637 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:03:57.965 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:03:57.965 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T14:03:57.965 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-05-06T14:03:57.965 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-05-06T14:03:57.965 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [], 2024-05-06T14:03:57.965 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "7/8 daemons upgraded", 2024-05-06T14:03:57.965 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "Currently upgrading osd daemons", 2024-05-06T14:03:57.965 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-05-06T14:03:57.965 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:03:58.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:58 smithi077 ceph-mon[84247]: from='client.64510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:03:58.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:58 smithi077 ceph-mon[84247]: from='client.54582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:03:58.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:58 smithi077 ceph-mon[84247]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 912 B/s rd, 0 op/s; 0 B/s, 6 objects/s recovering 2024-05-06T14:03:58.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:58 smithi077 ceph-mon[84247]: from='client.64522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:03:58.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:58 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/3738821458' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:03:58.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:58 smithi077 ceph-mon[89212]: from='client.64510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:03:58.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:58 smithi077 ceph-mon[89212]: from='client.54582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:03:58.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:58 smithi077 ceph-mon[89212]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 912 B/s rd, 0 op/s; 0 B/s, 6 objects/s recovering 2024-05-06T14:03:58.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:58 smithi077 ceph-mon[89212]: from='client.64522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:03:58.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:58 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/3738821458' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:03:58.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:58 smithi170 ceph-mon[65866]: from='client.64510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:03:58.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:58 smithi170 ceph-mon[65866]: from='client.54582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:03:58.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:58 smithi170 ceph-mon[65866]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 912 B/s rd, 0 op/s; 0 B/s, 6 objects/s recovering 2024-05-06T14:03:58.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:58 smithi170 ceph-mon[65866]: from='client.64522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:03:58.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:58 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/3738821458' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:03:59.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:03:59 smithi077 ceph-mon[84247]: from='client.64534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:03:59.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:03:59 smithi077 ceph-mon[89212]: from='client.64534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:03:59.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:03:59 smithi170 ceph-mon[65866]: from='client.64534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:04:00.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:00 smithi077 ceph-mon[84247]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s; 0 B/s, 5 objects/s recovering 2024-05-06T14:04:00.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:00 smithi077 ceph-mon[89212]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s; 0 B/s, 5 objects/s recovering 2024-05-06T14:04:00.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:00 smithi170 ceph-mon[65866]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s; 0 B/s, 5 objects/s recovering 2024-05-06T14:04:02.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:02 smithi077 ceph-mon[84247]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 768 B/s rd, 0 op/s; 0 B/s, 4 objects/s recovering 2024-05-06T14:04:02.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:02 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:02.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:02 smithi077 ceph-mon[89212]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 768 B/s rd, 0 op/s; 0 B/s, 4 objects/s recovering 2024-05-06T14:04:02.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:02 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:02.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:02 smithi170 ceph-mon[65866]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 768 B/s rd, 0 op/s; 0 B/s, 4 objects/s recovering 2024-05-06T14:04:02.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:02 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:03.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:03 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:04:03.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:03 smithi077 ceph-mon[89212]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 0 B/s, 4 objects/s recovering 2024-05-06T14:04:03.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:03 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:04:03.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:03 smithi077 ceph-mon[84247]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 0 B/s, 4 objects/s recovering 2024-05-06T14:04:03.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:03 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:04:03.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:03 smithi170 ceph-mon[65866]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 0 B/s, 4 objects/s recovering 2024-05-06T14:04:04.049 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:04:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:04:03] "GET /metrics HTTP/1.1" 200 38290 "" "Prometheus/2.43.0" 2024-05-06T14:04:06.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:06 smithi077 ceph-mon[84247]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 1 objects/s recovering 2024-05-06T14:04:06.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:06 smithi077 ceph-mon[89212]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 1 objects/s recovering 2024-05-06T14:04:06.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:06 smithi170 ceph-mon[65866]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 1 objects/s recovering 2024-05-06T14:04:08.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:08 smithi077 ceph-mon[89212]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:04:08.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:08 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-06T14:04:08.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:08 smithi077 ceph-mon[84247]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:04:08.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:08 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-06T14:04:08.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:08 smithi170 ceph-mon[65866]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:04:08.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:08 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-06T14:04:09.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:09 smithi077 ceph-mon[84247]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-06T14:04:09.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:09 smithi077 ceph-mon[84247]: Upgrade: osd.7 is safe to restart 2024-05-06T14:04:09.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:09 smithi077 ceph-mon[84247]: Upgrade: Updating osd.7 2024-05-06T14:04:09.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:09 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:09.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:09 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-06T14:04:09.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:09 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:04:09.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:09 smithi077 ceph-mon[84247]: Deploying daemon osd.7 on smithi170 2024-05-06T14:04:09.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:09 smithi077 ceph-mon[84247]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:04:09.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:09 smithi077 ceph-mon[89212]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-06T14:04:09.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:09 smithi077 ceph-mon[89212]: Upgrade: osd.7 is safe to restart 2024-05-06T14:04:09.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:09 smithi077 ceph-mon[89212]: Upgrade: Updating osd.7 2024-05-06T14:04:09.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:09 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:09.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:09 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-06T14:04:09.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:09 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:04:09.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:09 smithi077 ceph-mon[89212]: Deploying daemon osd.7 on smithi170 2024-05-06T14:04:09.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:09 smithi077 ceph-mon[89212]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:04:09.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:09 smithi170 ceph-mon[65866]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-05-06T14:04:09.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:09 smithi170 ceph-mon[65866]: Upgrade: osd.7 is safe to restart 2024-05-06T14:04:09.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:09 smithi170 ceph-mon[65866]: Upgrade: Updating osd.7 2024-05-06T14:04:09.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:09 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:09.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:09 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-06T14:04:09.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:09 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:04:09.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:09 smithi170 ceph-mon[65866]: Deploying daemon osd.7 on smithi170 2024-05-06T14:04:09.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:09 smithi170 ceph-mon[65866]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:04:11.380 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:11 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:04:11.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:11 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:04:11.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:11 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:04:12.495 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:12 smithi170 ceph-mon[65866]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:04:12.495 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:12 smithi170 systemd[1]: Stopping Ceph osd.7 for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:04:12.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:12 smithi077 ceph-mon[84247]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:04:12.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:12 smithi077 ceph-mon[89212]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:04:12.823 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:12 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[35425]: 2024-05-06T14:04:12.497+0000 7f973435b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:04:12.823 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:12 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[35425]: 2024-05-06T14:04:12.497+0000 7f973435b700 -1 osd.7 148 *** Got signal Terminated *** 2024-05-06T14:04:12.823 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:12 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[35425]: 2024-05-06T14:04:12.497+0000 7f973435b700 -1 osd.7 148 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T14:04:13.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:13 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:04:13.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:13 smithi077 ceph-mon[89212]: osd.7 marked itself down and dead 2024-05-06T14:04:13.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:13 smithi077 ceph-mon[89212]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:04:13.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:13 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:04:13.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:13 smithi077 ceph-mon[84247]: osd.7 marked itself down and dead 2024-05-06T14:04:13.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:13 smithi077 ceph-mon[84247]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:04:13.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:13 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:04:13.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:13 smithi170 ceph-mon[65866]: osd.7 marked itself down and dead 2024-05-06T14:04:13.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:13 smithi170 ceph-mon[65866]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:04:13.903 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:13 smithi170 podman[89986]: 2024-05-06 14:04:13.623412927 +0000 UTC m=+1.259354201 container died 8336b964be290244f74b9106a93dbd34c798376acf2501fa09b1857a3fc74925 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7, io.openshift.expose-services=, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, version=8, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, maintainer=Guillaume Abrioux , vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., release=754, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, ceph=True, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-05-06T14:04:14.049 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:04:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:04:13] "GET /metrics HTTP/1.1" 200 38290 "" "Prometheus/2.43.0" 2024-05-06T14:04:14.239 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:13 smithi170 podman[89986]: 2024-05-06 14:04:13.98116479 +0000 UTC m=+1.617106073 container cleanup 8336b964be290244f74b9106a93dbd34c798376acf2501fa09b1857a3fc74925 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7, maintainer=Guillaume Abrioux , release=754, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, vcs-type=git, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, vendor=Red Hat, Inc., io.openshift.expose-services=) 2024-05-06T14:04:14.239 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:13 smithi170 bash[89986]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7 2024-05-06T14:04:14.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:14 smithi077 ceph-mon[89212]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T14:04:14.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:14 smithi077 ceph-mon[89212]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-05-06T14:04:14.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:14 smithi077 ceph-mon[89212]: osdmap e149: 8 total, 7 up, 8 in 2024-05-06T14:04:14.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:14 smithi077 ceph-mon[84247]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T14:04:14.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:14 smithi077 ceph-mon[84247]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-05-06T14:04:14.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:14 smithi077 ceph-mon[84247]: osdmap e149: 8 total, 7 up, 8 in 2024-05-06T14:04:14.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:14 smithi170 ceph-mon[65866]: Health check failed: 1 osds down (OSD_DOWN) 2024-05-06T14:04:14.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:14 smithi170 ceph-mon[65866]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-05-06T14:04:14.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:14 smithi170 ceph-mon[65866]: osdmap e149: 8 total, 7 up, 8 in 2024-05-06T14:04:14.573 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:14 smithi170 podman[90005]: 2024-05-06 14:04:14.282599937 +0000 UTC m=+0.656712953 container remove 8336b964be290244f74b9106a93dbd34c798376acf2501fa09b1857a3fc74925 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., architecture=x86_64, version=8, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, name=centos-stream, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, RELEASE=HEAD, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, ceph=True) 2024-05-06T14:04:14.857 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:14 smithi170 podman[90099]: 2024-05-06 14:04:14.583786865 +0000 UTC m=+0.110964208 container create a409f6b42d8d416bea7c23ea0baaf6ca4a6556aa2b884df074e60fcd8e2fdf59 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-06T14:04:14.858 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:14 smithi170 podman[90099]: 2024-05-06 14:04:14.490888333 +0000 UTC m=+0.018065695 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:04:14.858 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:14 smithi170 podman[90099]: 2024-05-06 14:04:14.795064364 +0000 UTC m=+0.322241715 container init a409f6b42d8d416bea7c23ea0baaf6ca4a6556aa2b884df074e60fcd8e2fdf59 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7-deactivate, ceph=True, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T14:04:14.858 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:14 smithi170 podman[90099]: 2024-05-06 14:04:14.798539144 +0000 UTC m=+0.325716498 container start a409f6b42d8d416bea7c23ea0baaf6ca4a6556aa2b884df074e60fcd8e2fdf59 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-70ce953) 2024-05-06T14:04:14.858 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:14 smithi170 podman[90099]: 2024-05-06 14:04:14.861169563 +0000 UTC m=+0.388346912 container attach a409f6b42d8d416bea7c23ea0baaf6ca4a6556aa2b884df074e60fcd8e2fdf59 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T14:04:15.248 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:15 smithi170 podman[90099]: 2024-05-06 14:04:15.090209185 +0000 UTC m=+0.617386524 container died a409f6b42d8d416bea7c23ea0baaf6ca4a6556aa2b884df074e60fcd8e2fdf59 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-05-06T14:04:15.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:15 smithi077 ceph-mon[84247]: osdmap e150: 8 total, 7 up, 8 in 2024-05-06T14:04:15.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:15 smithi077 ceph-mon[84247]: pgmap v237: 161 pgs: 18 peering, 19 stale+active+clean, 124 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-05-06T14:04:15.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:15 smithi077 ceph-mon[89212]: osdmap e150: 8 total, 7 up, 8 in 2024-05-06T14:04:15.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:15 smithi077 ceph-mon[89212]: pgmap v237: 161 pgs: 18 peering, 19 stale+active+clean, 124 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-05-06T14:04:15.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:15 smithi170 ceph-mon[65866]: osdmap e150: 8 total, 7 up, 8 in 2024-05-06T14:04:15.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:15 smithi170 ceph-mon[65866]: pgmap v237: 161 pgs: 18 peering, 19 stale+active+clean, 124 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-05-06T14:04:16.071 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:15 smithi170 podman[90125]: 2024-05-06 14:04:15.640406461 +0000 UTC m=+0.544355709 container remove a409f6b42d8d416bea7c23ea0baaf6ca4a6556aa2b884df074e60fcd8e2fdf59 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T14:04:16.071 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:15 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.7.service: Deactivated successfully. 2024-05-06T14:04:16.071 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:15 smithi170 systemd[1]: Stopped Ceph osd.7 for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T14:04:16.071 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:15 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.7.service: Consumed 17.551s CPU time. 2024-05-06T14:04:16.071 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:15 smithi170 systemd[1]: Starting Ceph osd.7 for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:04:16.322 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:16 smithi170 ceph-mon[65866]: Health check failed: Reduced data availability: 3 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-05-06T14:04:16.323 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:16 smithi170 podman[90226]: 2024-05-06 14:04:16.074792944 +0000 UTC m=+0.111128929 container create 8824d8dd38bde50d9c76a48d872d290da7a52dd0ba714dc542d1a9481aa15d93 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7-activate, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS) 2024-05-06T14:04:16.323 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:16 smithi170 podman[90226]: 2024-05-06 14:04:15.982736247 +0000 UTC m=+0.019072246 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:04:16.323 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:16 smithi170 podman[90226]: 2024-05-06 14:04:16.310850225 +0000 UTC m=+0.347186210 container init 8824d8dd38bde50d9c76a48d872d290da7a52dd0ba714dc542d1a9481aa15d93 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7-activate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-06T14:04:16.323 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:16 smithi170 podman[90226]: 2024-05-06 14:04:16.314063262 +0000 UTC m=+0.350399251 container start 8824d8dd38bde50d9c76a48d872d290da7a52dd0ba714dc542d1a9481aa15d93 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7-activate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-06T14:04:16.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:16 smithi077 ceph-mon[84247]: Health check failed: Reduced data availability: 3 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-05-06T14:04:16.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:16 smithi077 ceph-mon[89212]: Health check failed: Reduced data availability: 3 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-05-06T14:04:16.756 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:16 smithi170 podman[90226]: 2024-05-06 14:04:16.410908704 +0000 UTC m=+0.447244704 container attach 8824d8dd38bde50d9c76a48d872d290da7a52dd0ba714dc542d1a9481aa15d93 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7-activate, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-06T14:04:17.073 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:16 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7-activate[90241]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-06T14:04:17.073 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:16 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7-activate[90241]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-05-06T14:04:17.073 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:16 smithi170 bash[90226]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-06T14:04:17.073 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:16 smithi170 bash[90226]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-05-06T14:04:17.073 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:16 smithi170 bash[90226]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-05-06T14:04:17.073 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:16 smithi170 bash[90226]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-05-06T14:04:17.073 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:16 smithi170 bash[90226]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-05-06T14:04:17.073 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:16 smithi170 bash[90226]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-06T14:04:17.073 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:16 smithi170 bash[90226]: --> ceph-volume raw activate successful for osd ID: 7 2024-05-06T14:04:17.073 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:16 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7-activate[90241]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-05-06T14:04:17.074 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:16 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7-activate[90241]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-05-06T14:04:17.074 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:16 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7-activate[90241]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-05-06T14:04:17.074 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:16 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7-activate[90241]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-06T14:04:17.074 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:16 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7-activate[90241]: --> ceph-volume raw activate successful for osd ID: 7 2024-05-06T14:04:17.074 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:16 smithi170 podman[90226]: 2024-05-06 14:04:16.784765216 +0000 UTC m=+0.821101204 container died 8824d8dd38bde50d9c76a48d872d290da7a52dd0ba714dc542d1a9481aa15d93 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7-activate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, GIT_CLEAN=True) 2024-05-06T14:04:17.420 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:17 smithi170 ceph-mon[65866]: pgmap v238: 161 pgs: 7 active+undersized+degraded, 18 peering, 9 stale+active+clean, 11 active+undersized, 116 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 34/627 objects degraded (5.423%) 2024-05-06T14:04:17.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:17 smithi077 ceph-mon[84247]: pgmap v238: 161 pgs: 7 active+undersized+degraded, 18 peering, 9 stale+active+clean, 11 active+undersized, 116 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 34/627 objects degraded (5.423%) 2024-05-06T14:04:17.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:17 smithi077 ceph-mon[89212]: pgmap v238: 161 pgs: 7 active+undersized+degraded, 18 peering, 9 stale+active+clean, 11 active+undersized, 116 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 34/627 objects degraded (5.423%) 2024-05-06T14:04:17.752 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:17 smithi170 podman[90373]: 2024-05-06 14:04:17.423933662 +0000 UTC m=+0.634046144 container remove 8824d8dd38bde50d9c76a48d872d290da7a52dd0ba714dc542d1a9481aa15d93 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7-activate, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2) 2024-05-06T14:04:17.752 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:17 smithi170 podman[90439]: 2024-05-06 14:04:17.543149583 +0000 UTC m=+0.018371207 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:04:17.752 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:17 smithi170 podman[90439]: 2024-05-06 14:04:17.650035924 +0000 UTC m=+0.125257579 container create 82e3dd2bbdd97faf327b4b2d80c22ffbb9e3d4e244d9fa234a57643c481a375a (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240430, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-05-06T14:04:18.073 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:17 smithi170 podman[90439]: 2024-05-06 14:04:17.893825633 +0000 UTC m=+0.369047245 container init 82e3dd2bbdd97faf327b4b2d80c22ffbb9e3d4e244d9fa234a57643c481a375a (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T14:04:18.073 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:17 smithi170 podman[90439]: 2024-05-06 14:04:17.896752535 +0000 UTC m=+0.371974149 container start 82e3dd2bbdd97faf327b4b2d80c22ffbb9e3d4e244d9fa234a57643c481a375a (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-06T14:04:18.073 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:17 smithi170 ceph-osd[90458]: -- 172.21.15.170:0/2840375955 <== mon.2 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55674277b680 con 0x556742745000 2024-05-06T14:04:18.073 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:17 smithi170 bash[90439]: 82e3dd2bbdd97faf327b4b2d80c22ffbb9e3d4e244d9fa234a57643c481a375a 2024-05-06T14:04:18.073 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:17 smithi170 systemd[1]: Started Ceph osd.7 for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T14:04:18.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:18 smithi077 ceph-mon[84247]: Health check failed: Degraded data redundancy: 34/627 objects degraded (5.423%), 7 pgs degraded (PG_DEGRADED) 2024-05-06T14:04:18.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:18 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:18.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:18 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:18.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:18 smithi077 ceph-mon[89212]: Health check failed: Degraded data redundancy: 34/627 objects degraded (5.423%), 7 pgs degraded (PG_DEGRADED) 2024-05-06T14:04:18.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:18 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:18.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:18 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:18.572 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:04:18.443+0000 7f3adfa927c0 -1 Falling back to public interface 2024-05-06T14:04:18.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:18 smithi170 ceph-mon[65866]: Health check failed: Degraded data redundancy: 34/627 objects degraded (5.423%), 7 pgs degraded (PG_DEGRADED) 2024-05-06T14:04:18.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:18 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:18.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:18 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:19.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:19 smithi077 ceph-mon[84247]: pgmap v239: 161 pgs: 22 active+undersized+degraded, 13 peering, 34 active+undersized, 92 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-05-06T14:04:19.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:19 smithi077 ceph-mon[89212]: pgmap v239: 161 pgs: 22 active+undersized+degraded, 13 peering, 34 active+undersized, 92 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-05-06T14:04:19.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:19 smithi170 ceph-mon[65866]: pgmap v239: 161 pgs: 22 active+undersized+degraded, 13 peering, 34 active+undersized, 92 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-05-06T14:04:21.440 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:21 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:04:21.159+0000 7f3adfa927c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-05-06T14:04:21.440 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:21 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:04:21.188+0000 7f3adfa927c0 -1 osd.7 148 log_to_monitors true 2024-05-06T14:04:21.440 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:21 smithi170 ceph-mon[65866]: pgmap v240: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-05-06T14:04:21.440 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:21 smithi170 ceph-mon[65866]: from='osd.7 [v2:172.21.15.170:6824/641497102,v1:172.21.15.170:6825/641497102]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-06T14:04:21.441 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:21 smithi170 ceph-mon[65866]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-06T14:04:21.441 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:21 smithi170 ceph-mon[65866]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-05-06T14:04:21.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:21 smithi077 ceph-mon[84247]: pgmap v240: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-05-06T14:04:21.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:21 smithi077 ceph-mon[84247]: from='osd.7 [v2:172.21.15.170:6824/641497102,v1:172.21.15.170:6825/641497102]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-06T14:04:21.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:21 smithi077 ceph-mon[84247]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-06T14:04:21.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:21 smithi077 ceph-mon[84247]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-05-06T14:04:21.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:21 smithi077 ceph-mon[89212]: pgmap v240: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-05-06T14:04:21.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:21 smithi077 ceph-mon[89212]: from='osd.7 [v2:172.21.15.170:6824/641497102,v1:172.21.15.170:6825/641497102]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-06T14:04:21.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:21 smithi077 ceph-mon[89212]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-06T14:04:21.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:21 smithi077 ceph-mon[89212]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-05-06T14:04:21.823 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:04:21 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:04:21.513+0000 7f3accee1700 -1 osd.7 148 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-06T14:04:22.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:22 smithi077 ceph-mon[84247]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-06T14:04:22.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:22 smithi077 ceph-mon[84247]: osdmap e151: 8 total, 7 up, 8 in 2024-05-06T14:04:22.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:22 smithi077 ceph-mon[84247]: from='osd.7 [v2:172.21.15.170:6824/641497102,v1:172.21.15.170:6825/641497102]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T14:04:22.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:22 smithi077 ceph-mon[84247]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T14:04:22.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:22 smithi077 ceph-mon[84247]: from='osd.7 ' entity='osd.7' 2024-05-06T14:04:22.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:22 smithi077 ceph-mon[89212]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-06T14:04:22.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:22 smithi077 ceph-mon[89212]: osdmap e151: 8 total, 7 up, 8 in 2024-05-06T14:04:22.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:22 smithi077 ceph-mon[89212]: from='osd.7 [v2:172.21.15.170:6824/641497102,v1:172.21.15.170:6825/641497102]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T14:04:22.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:22 smithi077 ceph-mon[89212]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T14:04:22.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:22 smithi077 ceph-mon[89212]: from='osd.7 ' entity='osd.7' 2024-05-06T14:04:22.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:22 smithi170 ceph-mon[65866]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-06T14:04:22.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:22 smithi170 ceph-mon[65866]: osdmap e151: 8 total, 7 up, 8 in 2024-05-06T14:04:22.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:22 smithi170 ceph-mon[65866]: from='osd.7 [v2:172.21.15.170:6824/641497102,v1:172.21.15.170:6825/641497102]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T14:04:22.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:22 smithi170 ceph-mon[65866]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-05-06T14:04:22.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:22 smithi170 ceph-mon[65866]: from='osd.7 ' entity='osd.7' 2024-05-06T14:04:23.735 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:23 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:04:23.735 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:23 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:23.735 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:23 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:23.736 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:23 smithi077 ceph-mon[84247]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T14:04:23.736 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:23 smithi077 ceph-mon[84247]: osd.7 [v2:172.21.15.170:6824/641497102,v1:172.21.15.170:6825/641497102] boot 2024-05-06T14:04:23.736 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:23 smithi077 ceph-mon[84247]: osdmap e152: 8 total, 8 up, 8 in 2024-05-06T14:04:23.736 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:23 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T14:04:23.736 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:23 smithi077 ceph-mon[84247]: Health check update: Degraded data redundancy: 84/627 objects degraded (13.397%), 27 pgs degraded (PG_DEGRADED) 2024-05-06T14:04:23.736 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:23 smithi077 ceph-mon[84247]: pgmap v243: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-05-06T14:04:23.736 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:23 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:04:23.736 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:23 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:23.736 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:23 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:23.736 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:23 smithi077 ceph-mon[89212]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T14:04:23.736 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:23 smithi077 ceph-mon[89212]: osd.7 [v2:172.21.15.170:6824/641497102,v1:172.21.15.170:6825/641497102] boot 2024-05-06T14:04:23.736 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:23 smithi077 ceph-mon[89212]: osdmap e152: 8 total, 8 up, 8 in 2024-05-06T14:04:23.737 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:23 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T14:04:23.737 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:23 smithi077 ceph-mon[89212]: Health check update: Degraded data redundancy: 84/627 objects degraded (13.397%), 27 pgs degraded (PG_DEGRADED) 2024-05-06T14:04:23.737 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:23 smithi077 ceph-mon[89212]: pgmap v243: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-05-06T14:04:23.822 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:23 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:04:23.823 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:23 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:23.823 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:23 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:23.823 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:23 smithi170 ceph-mon[65866]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-05-06T14:04:23.823 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:23 smithi170 ceph-mon[65866]: osd.7 [v2:172.21.15.170:6824/641497102,v1:172.21.15.170:6825/641497102] boot 2024-05-06T14:04:23.823 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:23 smithi170 ceph-mon[65866]: osdmap e152: 8 total, 8 up, 8 in 2024-05-06T14:04:23.823 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:23 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-06T14:04:23.823 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:23 smithi170 ceph-mon[65866]: Health check update: Degraded data redundancy: 84/627 objects degraded (13.397%), 27 pgs degraded (PG_DEGRADED) 2024-05-06T14:04:23.823 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:23 smithi170 ceph-mon[65866]: pgmap v243: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-05-06T14:04:24.048 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:04:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:04:23] "GET /metrics HTTP/1.1" 200 38290 "" "Prometheus/2.43.0" 2024-05-06T14:04:24.798 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:24 smithi077 ceph-mon[84247]: osdmap e153: 8 total, 8 up, 8 in 2024-05-06T14:04:24.799 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:24 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:24.799 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:24 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:24.799 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:24 smithi077 ceph-mon[89212]: osdmap e153: 8 total, 8 up, 8 in 2024-05-06T14:04:24.799 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:24 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:24.799 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:24 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:24.822 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:24 smithi170 ceph-mon[65866]: osdmap e153: 8 total, 8 up, 8 in 2024-05-06T14:04:24.822 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:24 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:24.822 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:24 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:25.798 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:25 smithi077 ceph-mon[84247]: pgmap v245: 161 pgs: 12 peering, 15 active+undersized+degraded, 28 active+undersized, 106 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 56/627 objects degraded (8.931%) 2024-05-06T14:04:25.799 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:25 smithi077 ceph-mon[89212]: pgmap v245: 161 pgs: 12 peering, 15 active+undersized+degraded, 28 active+undersized, 106 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 56/627 objects degraded (8.931%) 2024-05-06T14:04:25.822 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:25 smithi170 ceph-mon[65866]: pgmap v245: 161 pgs: 12 peering, 15 active+undersized+degraded, 28 active+undersized, 106 active+clean; 585 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 56/627 objects degraded (8.931%) 2024-05-06T14:04:27.322 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:27 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:27.323 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:27 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:04:27.323 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:27 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:27.323 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:27 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:27.323 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:27 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:04:27.323 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:27 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:04:27.323 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:27 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:27.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:27 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:27.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:27 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:04:27.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:27 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:27.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:27 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:27.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:27 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:04:27.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:27 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:04:27.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:27 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:27.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:27 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:27.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:27 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:04:27.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:27 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:27.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:27 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:27.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:27 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:04:27.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:27 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:04:27.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:27 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:28.344 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T14:04:28.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[89212]: pgmap v246: 161 pgs: 12 peering, 4 active+undersized+degraded, 8 active+undersized, 137 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 7/627 objects degraded (1.116%) 2024-05-06T14:04:28.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:04:28.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:28.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:28.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:28.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:28.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all osd 2024-05-06T14:04:28.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:28.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-05-06T14:04:28.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-05-06T14:04:28.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-05-06T14:04:28.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-05-06T14:04:28.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-05-06T14:04:28.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-05-06T14:04:28.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-05-06T14:04:28.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-05-06T14:04:28.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-05-06T14:04:28.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-05-06T14:04:28.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-05-06T14:04:28.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-05-06T14:04:28.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-05-06T14:04:28.550 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-05-06T14:04:28.551 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-05-06T14:04:28.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[84247]: pgmap v246: 161 pgs: 12 peering, 4 active+undersized+degraded, 8 active+undersized, 137 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 7/627 objects degraded (1.116%) 2024-05-06T14:04:28.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:04:28.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:28.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:28.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:28.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:28.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all osd 2024-05-06T14:04:28.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:28.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-05-06T14:04:28.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-05-06T14:04:28.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-05-06T14:04:28.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-05-06T14:04:28.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-05-06T14:04:28.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-05-06T14:04:28.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-05-06T14:04:28.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-05-06T14:04:28.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-05-06T14:04:28.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-05-06T14:04:28.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-05-06T14:04:28.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-05-06T14:04:28.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-05-06T14:04:28.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-05-06T14:04:28.552 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-05-06T14:04:28.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-05-06T14:04:28.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-05-06T14:04:28.553 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[89212]: Health check update: Degraded data redundancy: 56/627 objects degraded (8.931%), 15 pgs degraded (PG_DEGRADED) 2024-05-06T14:04:28.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-05-06T14:04:28.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-05-06T14:04:28.553 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:28 smithi077 ceph-mon[84247]: Health check update: Degraded data redundancy: 56/627 objects degraded (8.931%), 15 pgs degraded (PG_DEGRADED) 2024-05-06T14:04:28.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:28 smithi170 ceph-mon[65866]: pgmap v246: 161 pgs: 12 peering, 4 active+undersized+degraded, 8 active+undersized, 137 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 7/627 objects degraded (1.116%) 2024-05-06T14:04:28.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:04:28.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:28.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:28.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:28.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:28.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:28 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all osd 2024-05-06T14:04:28.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:28.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-05-06T14:04:28.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-05-06T14:04:28.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-05-06T14:04:28.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-05-06T14:04:28.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-05-06T14:04:28.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-05-06T14:04:28.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-05-06T14:04:28.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-05-06T14:04:28.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-05-06T14:04:28.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-05-06T14:04:28.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-05-06T14:04:28.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-05-06T14:04:28.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-05-06T14:04:28.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-05-06T14:04:28.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-05-06T14:04:28.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-05-06T14:04:28.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-05-06T14:04:28.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:28 smithi170 ceph-mon[65866]: Health check update: Degraded data redundancy: 56/627 objects degraded (8.931%), 15 pgs degraded (PG_DEGRADED) 2024-05-06T14:04:29.021 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T14:04:29.021 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (9m) 2m ago 31m 26.6M - 0.25.0 c8568f914cd2 997175e3e64e 2024-05-06T14:04:29.021 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (8m) 6s ago 31m 81.2M - 9.4.12 2bacad6d85d8 140bb9a2760a 2024-05-06T14:04:29.021 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (9m) 2m ago 30m 64.2M - 3.5 e1d6a67b021e 9353a303fb55 2024-05-06T14:04:29.021 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283,8765 running (10m) 6s ago 36m 452M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 125521be53a3 2024-05-06T14:04:29.021 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (20m) 2m ago 37m 509M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T14:04:29.021 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (6m) 2m ago 37m 50.5M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 5e50731ccbe8 2024-05-06T14:04:29.021 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (7m) 6s ago 36m 47.6M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 61cd48b3d0ff 2024-05-06T14:04:29.021 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (6m) 2m ago 36m 40.9M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 8a473cc2ffd1 2024-05-06T14:04:29.021 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (10m) 2m ago 32m 15.9M - 1.5.0 0da6a335fe13 67f45fa4968b 2024-05-06T14:04:29.021 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (10m) 6s ago 32m 16.3M - 1.5.0 0da6a335fe13 02081b71a5dd 2024-05-06T14:04:29.022 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (4m) 2m ago 35m 43.8M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 fbe876a933ed 2024-05-06T14:04:29.022 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (4m) 2m ago 35m 44.5M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 6f664e567c93 2024-05-06T14:04:29.022 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (3m) 2m ago 35m 43.8M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 a4c3266b87f8 2024-05-06T14:04:29.022 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (2m) 2m ago 34m 12.6M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 7c9e6dbacb88 2024-05-06T14:04:29.022 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (115s) 6s ago 34m 46.9M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 345a6ce2c7c3 2024-05-06T14:04:29.022 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (81s) 6s ago 33m 44.9M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 fa8ea71417a6 2024-05-06T14:04:29.022 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (46s) 6s ago 33m 46.7M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 1fbae25547b9 2024-05-06T14:04:29.022 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (11s) 6s ago 33m 12.6M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 82e3dd2bbdd9 2024-05-06T14:04:29.022 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (9m) 6s ago 32m 73.0M - 2.43.0 a07b618ecd1d 10d5c513bb9b 2024-05-06T14:04:29.022 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (30m) 2m ago 30m 79.4M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T14:04:29.022 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (30m) 6s ago 30m 79.5M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T14:04:29.299 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: Upgrade: Setting require_osd_release to 19 squid 2024-05-06T14:04:29.299 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-05-06T14:04:29.299 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-05-06T14:04:29.299 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: osdmap e154: 8 total, 8 up, 8 in 2024-05-06T14:04:29.299 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.299 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:29.299 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.299 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:29.300 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.300 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.300 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:29.300 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.300 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:29.300 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.300 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:29.300 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.300 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.300 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.300 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.300 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.300 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.300 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-05-06T14:04:29.300 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-05-06T14:04:29.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:04:29.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-05-06T14:04:29.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-05-06T14:04:29.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-05-06T14:04:29.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-05-06T14:04:29.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-05-06T14:04:29.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-05-06T14:04:29.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-05-06T14:04:29.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-05-06T14:04:29.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-05-06T14:04:29.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-05-06T14:04:29.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:04:29.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-05-06T14:04:29.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-05-06T14:04:29.301 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-05-06T14:04:29.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-05-06T14:04:29.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-05-06T14:04:29.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-05-06T14:04:29.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-05-06T14:04:29.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:04:29.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:04:29.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:04:29.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:04:29.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:04:29.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:04:29.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-05-06T14:04:29.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-05-06T14:04:29.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: Upgrade: Setting require_osd_release to 19 squid 2024-05-06T14:04:29.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-05-06T14:04:29.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-05-06T14:04:29.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: osdmap e154: 8 total, 8 up, 8 in 2024-05-06T14:04:29.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:29.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:29.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:29.303 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:29.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:29.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-05-06T14:04:29.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-05-06T14:04:29.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:04:29.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-05-06T14:04:29.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-05-06T14:04:29.304 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-05-06T14:04:29.305 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-05-06T14:04:29.305 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:04:29.305 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:04:29.305 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:29.305 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:04:29.305 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:04:29.305 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:29.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-05-06T14:04:29.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-05-06T14:04:29.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-05-06T14:04:29.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-05-06T14:04:29.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-05-06T14:04:29.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-05-06T14:04:29.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:04:29.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-05-06T14:04:29.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-05-06T14:04:29.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-05-06T14:04:29.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-05-06T14:04:29.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-05-06T14:04:29.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-05-06T14:04:29.306 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-05-06T14:04:29.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:04:29.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:04:29.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:04:29.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:04:29.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:04:29.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:04:29.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-05-06T14:04:29.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-05-06T14:04:29.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:04:29.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:04:29.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:29.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:04:29.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:04:29.307 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:29.420 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:04:29.420 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-05-06T14:04:29.421 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T14:04:29.421 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:04:29.421 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-05-06T14:04:29.421 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T14:04:29.421 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:04:29.421 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-05-06T14:04:29.421 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 8 2024-05-06T14:04:29.421 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:04:29.421 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw": { 2024-05-06T14:04:29.421 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T14:04:29.421 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:04:29.421 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-05-06T14:04:29.421 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-05-06T14:04:29.422 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 13 2024-05-06T14:04:29.422 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T14:04:29.422 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:04:29.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: Upgrade: Setting require_osd_release to 19 squid 2024-05-06T14:04:29.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-05-06T14:04:29.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-05-06T14:04:29.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: osdmap e154: 8 total, 8 up, 8 in 2024-05-06T14:04:29.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:29.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:29.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:29.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:29.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:29.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:29.574 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-05-06T14:04:29.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-05-06T14:04:29.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:04:29.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-05-06T14:04:29.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-05-06T14:04:29.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-05-06T14:04:29.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-05-06T14:04:29.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-05-06T14:04:29.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-05-06T14:04:29.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-05-06T14:04:29.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-05-06T14:04:29.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-05-06T14:04:29.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-05-06T14:04:29.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:04:29.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-05-06T14:04:29.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-05-06T14:04:29.575 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-05-06T14:04:29.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-05-06T14:04:29.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-05-06T14:04:29.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-05-06T14:04:29.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-05-06T14:04:29.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:04:29.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:04:29.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:04:29.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:04:29.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:04:29.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:04:29.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-05-06T14:04:29.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-05-06T14:04:29.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:04:29.576 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:04:29.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:29.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:04:29.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:04:29.577 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:29.752 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:04:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": null, 2024-05-06T14:04:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": false, 2024-05-06T14:04:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "", 2024-05-06T14:04:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [], 2024-05-06T14:04:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": null, 2024-05-06T14:04:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "", 2024-05-06T14:04:29.753 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-05-06T14:04:29.753 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:04:30.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:30 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all mds 2024-05-06T14:04:30.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:30 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all rbd-mirror 2024-05-06T14:04:30.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:30 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all ceph-exporter 2024-05-06T14:04:30.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:30 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all nfs 2024-05-06T14:04:30.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:30 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all nvmeof 2024-05-06T14:04:30.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:30 smithi077 ceph-mon[89212]: Upgrade: Finalizing container_image settings 2024-05-06T14:04:30.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:30 smithi077 ceph-mon[89212]: from='client.54612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:04:30.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:30 smithi077 ceph-mon[89212]: Upgrade: Complete! 2024-05-06T14:04:30.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:30 smithi077 ceph-mon[89212]: from='client.64552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:04:30.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:30 smithi077 ceph-mon[89212]: from='client.54624 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:04:30.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:30 smithi077 ceph-mon[89212]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-05-06T14:04:30.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:30 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/1409951461' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:30.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:30 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all mds 2024-05-06T14:04:30.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:30 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all rbd-mirror 2024-05-06T14:04:30.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:30 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all ceph-exporter 2024-05-06T14:04:30.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:30 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all nfs 2024-05-06T14:04:30.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:30 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all nvmeof 2024-05-06T14:04:30.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:30 smithi077 ceph-mon[84247]: Upgrade: Finalizing container_image settings 2024-05-06T14:04:30.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:30 smithi077 ceph-mon[84247]: from='client.54612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:04:30.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:30 smithi077 ceph-mon[84247]: Upgrade: Complete! 2024-05-06T14:04:30.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:30 smithi077 ceph-mon[84247]: from='client.64552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:04:30.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:30 smithi077 ceph-mon[84247]: from='client.54624 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:04:30.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:30 smithi077 ceph-mon[84247]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-05-06T14:04:30.551 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:30 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/1409951461' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:30.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:30 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all mds 2024-05-06T14:04:30.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:30 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all rbd-mirror 2024-05-06T14:04:30.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:30 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all ceph-exporter 2024-05-06T14:04:30.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:30 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all nfs 2024-05-06T14:04:30.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:30 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all nvmeof 2024-05-06T14:04:30.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:30 smithi170 ceph-mon[65866]: Upgrade: Finalizing container_image settings 2024-05-06T14:04:30.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:30 smithi170 ceph-mon[65866]: from='client.54612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:04:30.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:30 smithi170 ceph-mon[65866]: Upgrade: Complete! 2024-05-06T14:04:30.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:30 smithi170 ceph-mon[65866]: from='client.64552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:04:30.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:30 smithi170 ceph-mon[65866]: from='client.54624 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:04:30.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:30 smithi170 ceph-mon[65866]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-05-06T14:04:30.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:30 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/1409951461' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:04:31.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:31 smithi077 ceph-mon[89212]: from='client.54636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:04:31.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:31 smithi077 ceph-mon[89212]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/627 objects degraded (1.116%), 4 pgs degraded) 2024-05-06T14:04:31.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:31 smithi077 ceph-mon[89212]: Cluster is now healthy 2024-05-06T14:04:31.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:31 smithi077 ceph-mon[84247]: from='client.54636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:04:31.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:31 smithi077 ceph-mon[84247]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/627 objects degraded (1.116%), 4 pgs degraded) 2024-05-06T14:04:31.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:31 smithi077 ceph-mon[84247]: Cluster is now healthy 2024-05-06T14:04:31.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:31 smithi170 ceph-mon[65866]: from='client.54636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:04:31.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:31 smithi170 ceph-mon[65866]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/627 objects degraded (1.116%), 4 pgs degraded) 2024-05-06T14:04:31.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:31 smithi170 ceph-mon[65866]: Cluster is now healthy 2024-05-06T14:04:32.449 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:32 smithi077 ceph-mon[84247]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:04:32.449 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:32 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:32.450 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:32 smithi077 ceph-mon[89212]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:04:32.450 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:32 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:32.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:32 smithi170 ceph-mon[65866]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:04:32.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:32 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:33.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:33 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:04:33.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:33 smithi077 ceph-mon[89212]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 530 B/s rd, 0 op/s 2024-05-06T14:04:33.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:33 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:04:33.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:33 smithi077 ceph-mon[84247]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 530 B/s rd, 0 op/s 2024-05-06T14:04:33.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:33 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:04:33.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:33 smithi170 ceph-mon[65866]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 530 B/s rd, 0 op/s 2024-05-06T14:04:34.048 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:04:33 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:04:33] "GET /metrics HTTP/1.1" 200 38231 "" "Prometheus/2.43.0" 2024-05-06T14:04:35.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:35 smithi077 ceph-mon[84247]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T14:04:35.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:35 smithi077 ceph-mon[89212]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T14:04:35.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:35 smithi170 ceph-mon[65866]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T14:04:37.798 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:37 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:37.798 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:37 smithi077 ceph-mon[89212]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T14:04:37.799 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:37 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:37.799 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:37 smithi077 ceph-mon[84247]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T14:04:37.822 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:37 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:37.822 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:37 smithi170 ceph-mon[65866]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-05-06T14:04:39.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:39 smithi077 ceph-mon[84247]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 op/s 2024-05-06T14:04:39.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:39 smithi077 ceph-mon[89212]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 op/s 2024-05-06T14:04:39.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:39 smithi170 ceph-mon[65866]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 op/s 2024-05-06T14:04:42.466 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:42 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:42.466 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:42 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:04:42.467 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:42 smithi077 ceph-mon[84247]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:04:42.467 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:42 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:42.467 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:42 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:04:42.467 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:42 smithi077 ceph-mon[89212]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:04:42.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:42 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:04:42.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:42 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:04:42.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:42 smithi170 ceph-mon[65866]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:04:43.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:43 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:04:43.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:43 smithi077 ceph-mon[84247]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:04:43.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:43 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:04:43.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:43 smithi077 ceph-mon[89212]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:04:43.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:43 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:04:43.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:43 smithi170 ceph-mon[65866]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:04:44.048 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:04:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:04:43] "GET /metrics HTTP/1.1" 200 38376 "" "Prometheus/2.43.0" 2024-05-06T14:04:45.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:45 smithi077 ceph-mon[84247]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:04:45.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:45 smithi077 ceph-mon[89212]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:04:45.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:45 smithi170 ceph-mon[65866]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:04:47.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:47 smithi077 ceph-mon[84247]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:04:47.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:47 smithi077 ceph-mon[89212]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:04:47.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:47 smithi170 ceph-mon[65866]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:04:49.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:49 smithi077 ceph-mon[84247]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:04:49.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:49 smithi077 ceph-mon[89212]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:04:49.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:49 smithi170 ceph-mon[65866]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:04:51.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:51 smithi077 ceph-mon[84247]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:04:51.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:51 smithi077 ceph-mon[89212]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:04:51.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:51 smithi170 ceph-mon[65866]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:04:53.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:53 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:04:53.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:53 smithi077 ceph-mon[89212]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:04:53.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:53 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:04:53.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:53 smithi077 ceph-mon[84247]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:04:53.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:53 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:04:53.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:53 smithi170 ceph-mon[65866]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:04:54.048 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:04:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:04:53] "GET /metrics HTTP/1.1" 200 38376 "" "Prometheus/2.43.0" 2024-05-06T14:04:55.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:55 smithi077 ceph-mon[84247]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:04:55.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:55 smithi077 ceph-mon[89212]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:04:55.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:55 smithi170 ceph-mon[65866]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:04:56.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:56 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:04:56.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:56 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:04:56.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:56 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:04:57.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:57 smithi077 ceph-mon[84247]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:04:57.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:57 smithi077 ceph-mon[89212]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:04:57.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:57 smithi170 ceph-mon[65866]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:04:59.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:04:59 smithi077 ceph-mon[84247]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:04:59.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:04:59 smithi077 ceph-mon[89212]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:04:59.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:04:59 smithi170 ceph-mon[65866]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:05:00.427 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:00 smithi077 ceph-mon[84247]: from='client.54642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:00.427 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:00 smithi077 ceph-mon[89212]: from='client.54642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:00.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:00 smithi170 ceph-mon[65866]: from='client.54642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:00.830 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch ps' 2024-05-06T14:05:01.448 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:01 smithi077 ceph-mon[84247]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:05:01.449 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:01 smithi077 ceph-mon[89212]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:05:01.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:01 smithi170 ceph-mon[65866]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:05:02.779 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T14:05:02.779 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (9m) 2m ago 32m 26.6M - 0.25.0 c8568f914cd2 997175e3e64e 2024-05-06T14:05:02.779 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (9m) 40s ago 31m 81.2M - 9.4.12 2bacad6d85d8 140bb9a2760a 2024-05-06T14:05:02.780 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (10m) 2m ago 31m 64.2M - 3.5 e1d6a67b021e 9353a303fb55 2024-05-06T14:05:02.780 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283,8765 running (11m) 40s ago 36m 452M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 125521be53a3 2024-05-06T14:05:02.780 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (20m) 2m ago 38m 509M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T14:05:02.780 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (7m) 2m ago 38m 50.5M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 5e50731ccbe8 2024-05-06T14:05:02.780 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (8m) 40s ago 36m 47.6M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 61cd48b3d0ff 2024-05-06T14:05:02.780 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (7m) 2m ago 37m 40.9M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 8a473cc2ffd1 2024-05-06T14:05:02.780 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (10m) 2m ago 32m 15.9M - 1.5.0 0da6a335fe13 67f45fa4968b 2024-05-06T14:05:02.780 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (10m) 40s ago 32m 16.3M - 1.5.0 0da6a335fe13 02081b71a5dd 2024-05-06T14:05:02.780 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (5m) 2m ago 36m 43.8M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 fbe876a933ed 2024-05-06T14:05:02.780 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (4m) 2m ago 36m 44.5M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 6f664e567c93 2024-05-06T14:05:02.780 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (3m) 2m ago 35m 43.8M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 a4c3266b87f8 2024-05-06T14:05:02.780 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (3m) 2m ago 35m 12.6M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 7c9e6dbacb88 2024-05-06T14:05:02.780 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (2m) 40s ago 34m 46.9M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 345a6ce2c7c3 2024-05-06T14:05:02.780 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (115s) 40s ago 34m 44.9M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 fa8ea71417a6 2024-05-06T14:05:02.780 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (80s) 40s ago 34m 46.7M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 1fbae25547b9 2024-05-06T14:05:02.780 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (45s) 40s ago 33m 12.6M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 82e3dd2bbdd9 2024-05-06T14:05:02.781 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (10m) 40s ago 33m 73.0M - 2.43.0 a07b618ecd1d 10d5c513bb9b 2024-05-06T14:05:02.781 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (31m) 2m ago 31m 79.4M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T14:05:02.781 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (31m) 40s ago 31m 79.5M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T14:05:03.435 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:03 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:05:03.435 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:03 smithi077 ceph-mon[84247]: from='client.54648 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:03.435 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:03 smithi077 ceph-mon[84247]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:05:03.435 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:03 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:05:03.435 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:03 smithi077 ceph-mon[89212]: from='client.54648 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:03.436 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:03 smithi077 ceph-mon[89212]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:05:03.461 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 1'"'"'' 2024-05-06T14:05:03.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:03 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:05:03.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:03 smithi170 ceph-mon[65866]: from='client.54648 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:03.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:03 smithi170 ceph-mon[65866]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:05:03.997 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:05:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:05:03] "GET /metrics HTTP/1.1" 200 38377 "" "Prometheus/2.43.0" 2024-05-06T14:05:05.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:05 smithi077 ceph-mon[84247]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:05:05.549 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:05 smithi077 ceph-mon[89212]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:05:05.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:05 smithi170 ceph-mon[65866]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:05:05.683 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T14:05:06.390 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph versions | jq -e '"'"'.osd | keys'"'"' | grep $sha1' 2024-05-06T14:05:06.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:06 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/1154523743' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:06.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:06 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/1154523743' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:06.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:06 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/1154523743' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:07.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:07 smithi077 ceph-mon[84247]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:05:07.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:07 smithi077 ceph-mon[89212]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:05:07.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:07 smithi170 ceph-mon[65866]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:05:08.534 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)" 2024-05-06T14:05:08.798 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:08 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/3174491139' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:08.798 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:08 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/3174491139' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:09.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:08 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/3174491139' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:09.165 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch upgrade status' 2024-05-06T14:05:09.798 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:09 smithi077 ceph-mon[84247]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:05:09.798 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:09 smithi077 ceph-mon[89212]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:05:10.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:09 smithi170 ceph-mon[65866]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:05:11.367 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:05:11.367 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": null, 2024-05-06T14:05:11.367 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": false, 2024-05-06T14:05:11.367 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "", 2024-05-06T14:05:11.367 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [], 2024-05-06T14:05:11.367 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": null, 2024-05-06T14:05:11.368 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "", 2024-05-06T14:05:11.368 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-05-06T14:05:11.368 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:05:11.368 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:11 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:05:11.368 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:11 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:05:11.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:11 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:05:11.979 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph health detail' 2024-05-06T14:05:12.280 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:12 smithi077 ceph-mon[84247]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:05:12.280 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:12 smithi077 ceph-mon[89212]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:05:12.473 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:12 smithi170 ceph-mon[65866]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:05:13.298 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:13 smithi077 ceph-mon[84247]: from='client.64600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:13.298 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:13 smithi077 ceph-mon[89212]: from='client.64600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:13.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:13 smithi170 ceph-mon[65866]: from='client.64600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:14.048 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:05:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:05:13] "GET /metrics HTTP/1.1" 200 38377 "" "Prometheus/2.43.0" 2024-05-06T14:05:14.113 INFO:teuthology.orchestra.run.smithi077.stdout:HEALTH_OK 2024-05-06T14:05:14.412 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:14 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:05:14.412 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:14 smithi077 ceph-mon[84247]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:05:14.412 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:14 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/2011443337' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T14:05:14.412 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:14 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:05:14.412 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:14 smithi077 ceph-mon[89212]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:05:14.413 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:14 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/2011443337' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T14:05:14.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:14 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:05:14.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:14 smithi170 ceph-mon[65866]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:05:14.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:14 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/2011443337' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T14:05:14.889 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --services rgw.foo' 2024-05-06T14:05:16.430 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:16 smithi077 ceph-mon[84247]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:05:16.431 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:16 smithi077 ceph-mon[89212]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:05:16.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:16 smithi170 ceph-mon[65866]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:05:18.405 INFO:teuthology.orchestra.run.smithi077.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:05:18.468 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:18 smithi077 ceph-mon[84247]: from='client.54678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:18.469 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:18 smithi077 ceph-mon[84247]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:05:18.469 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:18 smithi077 ceph-mon[89212]: from='client.54678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:18.469 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:18 smithi077 ceph-mon[89212]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:05:18.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:18 smithi170 ceph-mon[65866]: from='client.54678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:18.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:18 smithi170 ceph-mon[65866]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:05:19.032 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-05-06T14:05:19.798 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:19 smithi077 ceph-mon[84247]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:05:19.798 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:19 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:19.798 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:19 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:05:19.798 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:19 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:05:19.798 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:19 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:19.798 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:19 smithi077 ceph-mon[84247]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:05:19.798 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:19 smithi077 ceph-mon[84247]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:05:19.798 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:19 smithi077 ceph-mon[89212]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:05:19.798 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:19 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:19.799 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:19 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:05:19.799 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:19 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:05:19.799 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:19 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:19.799 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:19 smithi077 ceph-mon[89212]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:05:19.799 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:19 smithi077 ceph-mon[89212]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:05:19.821 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:19 smithi170 ceph-mon[65866]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:05:19.822 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:19 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:19.822 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:19 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:05:19.822 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:19 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:05:19.822 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:19 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:19.822 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:19 smithi170 ceph-mon[65866]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:05:19.822 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:19 smithi170 ceph-mon[65866]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:05:21.449 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:21.449 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[89212]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (squid) 2024-05-06T14:05:21.450 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[89212]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T14:05:21.450 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:05:21.450 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:21.450 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all mgr 2024-05-06T14:05:21.450 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:21.450 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:21.450 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all mon 2024-05-06T14:05:21.450 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:21.450 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:21.450 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all crash 2024-05-06T14:05:21.450 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:21.450 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:21.451 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all osd 2024-05-06T14:05:21.451 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:21.451 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:21.451 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all mds 2024-05-06T14:05:21.451 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:21.451 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[89212]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:05:21.451 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:21.451 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[84247]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (squid) 2024-05-06T14:05:21.451 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[84247]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T14:05:21.452 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:05:21.452 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:21.452 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all mgr 2024-05-06T14:05:21.452 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:21.452 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:21.452 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all mon 2024-05-06T14:05:21.452 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:21.452 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:21.452 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all crash 2024-05-06T14:05:21.452 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:21.452 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:21.452 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all osd 2024-05-06T14:05:21.453 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:21.453 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:21.453 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all mds 2024-05-06T14:05:21.453 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:21.453 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:21 smithi077 ceph-mon[84247]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:05:21.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:21 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:21.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:21 smithi170 ceph-mon[65866]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (squid) 2024-05-06T14:05:21.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:21 smithi170 ceph-mon[65866]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T14:05:21.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:21 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:05:21.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:21 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:21.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:21 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all mgr 2024-05-06T14:05:21.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:21 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:21.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:21 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:21.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:21 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all mon 2024-05-06T14:05:21.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:21 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:21.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:21 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:21.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:21 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all crash 2024-05-06T14:05:21.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:21 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:21.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:21 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:21.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:21 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all osd 2024-05-06T14:05:21.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:21 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:21.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:21 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:21.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:21 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all mds 2024-05-06T14:05:21.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:21 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:21.573 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:21 smithi170 ceph-mon[65866]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:05:22.111 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T14:05:22.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:22 smithi077 ceph-mon[84247]: from='client.64618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:22.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:22 smithi077 ceph-mon[89212]: from='client.64618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:22.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:22 smithi170 ceph-mon[65866]: from='client.64618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:22.785 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T14:05:22.785 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (10m) 3m ago 32m 26.6M - 0.25.0 c8568f914cd2 997175e3e64e 2024-05-06T14:05:22.785 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (9m) 60s ago 32m 81.2M - 9.4.12 2bacad6d85d8 140bb9a2760a 2024-05-06T14:05:22.785 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (10m) 3m ago 31m 64.2M - 3.5 e1d6a67b021e 9353a303fb55 2024-05-06T14:05:22.785 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283,8765 running (11m) 60s ago 37m 452M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 125521be53a3 2024-05-06T14:05:22.785 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (21m) 3m ago 38m 509M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T14:05:22.785 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (7m) 3m ago 38m 50.5M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 5e50731ccbe8 2024-05-06T14:05:22.785 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (8m) 60s ago 37m 47.6M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 61cd48b3d0ff 2024-05-06T14:05:22.785 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (7m) 3m ago 37m 40.9M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 8a473cc2ffd1 2024-05-06T14:05:22.786 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (11m) 3m ago 33m 15.9M - 1.5.0 0da6a335fe13 67f45fa4968b 2024-05-06T14:05:22.786 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (11m) 60s ago 33m 16.3M - 1.5.0 0da6a335fe13 02081b71a5dd 2024-05-06T14:05:22.786 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (5m) 3m ago 36m 43.8M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 fbe876a933ed 2024-05-06T14:05:22.786 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (5m) 3m ago 36m 44.5M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 6f664e567c93 2024-05-06T14:05:22.786 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (4m) 3m ago 35m 43.8M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 a4c3266b87f8 2024-05-06T14:05:22.786 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (3m) 3m ago 35m 12.6M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 7c9e6dbacb88 2024-05-06T14:05:22.786 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (2m) 60s ago 35m 46.9M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 345a6ce2c7c3 2024-05-06T14:05:22.786 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (2m) 60s ago 34m 44.9M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 fa8ea71417a6 2024-05-06T14:05:22.786 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (100s) 60s ago 34m 46.7M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 1fbae25547b9 2024-05-06T14:05:22.786 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (65s) 60s ago 34m 12.6M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 82e3dd2bbdd9 2024-05-06T14:05:22.786 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (10m) 60s ago 33m 73.0M - 2.43.0 a07b618ecd1d 10d5c513bb9b 2024-05-06T14:05:22.786 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (31m) 3m ago 31m 79.4M - 17.2.0 e1d6a67b021e 66f82976f9f6 2024-05-06T14:05:22.787 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (31m) 60s ago 31m 79.5M - 17.2.0 e1d6a67b021e fd8beb411378 2024-05-06T14:05:23.183 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:05:23.184 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-05-06T14:05:23.184 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T14:05:23.184 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:05:23.184 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-05-06T14:05:23.184 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T14:05:23.184 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:05:23.184 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-05-06T14:05:23.184 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 8 2024-05-06T14:05:23.184 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:05:23.184 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw": { 2024-05-06T14:05:23.184 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-05-06T14:05:23.184 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:05:23.184 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-05-06T14:05:23.185 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-05-06T14:05:23.185 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 13 2024-05-06T14:05:23.185 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T14:05:23.185 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:05:23.533 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:05:23.534 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T14:05:23.534 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-05-06T14:05:23.534 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading daemons in service(s) rgw.foo", 2024-05-06T14:05:23.534 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [], 2024-05-06T14:05:23.534 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "0/2 daemons upgraded", 2024-05-06T14:05:23.534 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "Currently upgrading rgw daemons", 2024-05-06T14:05:23.534 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-05-06T14:05:23.535 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:05:23.797 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:05:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:05:23] "GET /metrics HTTP/1.1" 200 38377 "" "Prometheus/2.43.0" 2024-05-06T14:05:24.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:24 smithi077 ceph-mon[89212]: from='client.64624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:24.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:24 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:05:24.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:24 smithi077 ceph-mon[89212]: from='client.64630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:24.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:24 smithi077 ceph-mon[89212]: Upgrade: Updating rgw.foo.smithi077.xctpgy (1/2) 2024-05-06T14:05:24.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:24 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:24.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:24 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi077.xctpgy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T14:05:24.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:24 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:05:24.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:24 smithi077 ceph-mon[89212]: Deploying daemon rgw.foo.smithi077.xctpgy on smithi077 2024-05-06T14:05:24.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:24 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/3828475079' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:24.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:24 smithi077 ceph-mon[89212]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:05:24.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:24 smithi077 ceph-mon[84247]: from='client.64624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:24.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:24 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:05:24.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:24 smithi077 ceph-mon[84247]: from='client.64630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:24.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:24 smithi077 ceph-mon[84247]: Upgrade: Updating rgw.foo.smithi077.xctpgy (1/2) 2024-05-06T14:05:24.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:24 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:24.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:24 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi077.xctpgy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T14:05:24.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:24 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:05:24.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:24 smithi077 ceph-mon[84247]: Deploying daemon rgw.foo.smithi077.xctpgy on smithi077 2024-05-06T14:05:24.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:24 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/3828475079' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:24.550 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:24 smithi077 ceph-mon[84247]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:05:24.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:24 smithi170 ceph-mon[65866]: from='client.64624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:24.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:24 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:05:24.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:24 smithi170 ceph-mon[65866]: from='client.64630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:24.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:24 smithi170 ceph-mon[65866]: Upgrade: Updating rgw.foo.smithi077.xctpgy (1/2) 2024-05-06T14:05:24.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:24 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:24.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:24 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi077.xctpgy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T14:05:24.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:24 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:05:24.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:24 smithi170 ceph-mon[65866]: Deploying daemon rgw.foo.smithi077.xctpgy on smithi077 2024-05-06T14:05:24.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:24 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/3828475079' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:24.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:24 smithi170 ceph-mon[65866]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:05:25.290 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:25 smithi077 ceph-mon[84247]: from='client.54708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:25.290 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:25 smithi077 ceph-mon[89212]: from='client.54708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:25.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:25 smithi170 ceph-mon[65866]: from='client.54708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:26.298 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:26 smithi077 ceph-mon[84247]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:05:26.298 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:26 smithi077 ceph-mon[89212]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:05:26.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:26 smithi170 ceph-mon[65866]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:05:27.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:27 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:05:27.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:27 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:05:27.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:27 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:05:28.204 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:28 smithi170 ceph-mon[65866]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T14:05:28.205 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:28.205 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:28 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:28.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:28 smithi077 ceph-mon[84247]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T14:05:28.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:28.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:28 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:28.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:28 smithi077 ceph-mon[89212]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-05-06T14:05:28.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:28.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:28 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:30.124 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:29 smithi170 ceph-mon[65866]: Upgrade: Updating rgw.foo.smithi170.rackmk (2/2) 2024-05-06T14:05:30.124 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:30.124 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi170.rackmk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T14:05:30.124 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:29 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:05:30.124 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:29 smithi170 ceph-mon[65866]: Deploying daemon rgw.foo.smithi170.rackmk on smithi170 2024-05-06T14:05:30.124 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:29 smithi170 ceph-mon[65866]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 85 B/s wr, 28 op/s 2024-05-06T14:05:30.298 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:29 smithi077 ceph-mon[84247]: Upgrade: Updating rgw.foo.smithi170.rackmk (2/2) 2024-05-06T14:05:30.298 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:30.298 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi170.rackmk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T14:05:30.298 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:29 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:05:30.298 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:29 smithi077 ceph-mon[84247]: Deploying daemon rgw.foo.smithi170.rackmk on smithi170 2024-05-06T14:05:30.298 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:29 smithi077 ceph-mon[84247]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 85 B/s wr, 28 op/s 2024-05-06T14:05:30.298 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:29 smithi077 ceph-mon[89212]: Upgrade: Updating rgw.foo.smithi170.rackmk (2/2) 2024-05-06T14:05:30.298 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:30.299 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi170.rackmk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-06T14:05:30.299 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:29 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:05:30.299 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:29 smithi077 ceph-mon[89212]: Deploying daemon rgw.foo.smithi170.rackmk on smithi170 2024-05-06T14:05:30.299 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:29 smithi077 ceph-mon[89212]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 85 B/s wr, 28 op/s 2024-05-06T14:05:31.487 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:31 smithi170 ceph-mon[65866]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 85 B/s wr, 40 op/s 2024-05-06T14:05:31.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:31 smithi077 ceph-mon[84247]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 85 B/s wr, 40 op/s 2024-05-06T14:05:31.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:31 smithi077 ceph-mon[89212]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 85 B/s wr, 40 op/s 2024-05-06T14:05:33.962 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:33 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:05:33.962 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:33 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:33.962 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:33 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:33.962 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:33 smithi170 ceph-mon[65866]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-05-06T14:05:34.026 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:05:33 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:05:33] "GET /metrics HTTP/1.1" 200 38375 "" "Prometheus/2.43.0" 2024-05-06T14:05:34.026 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:33 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:05:34.026 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:33 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:34.026 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:33 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:34.026 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:33 smithi077 ceph-mon[84247]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-05-06T14:05:34.027 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:33 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:05:34.027 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:33 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:34.027 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:33 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:34.027 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:33 smithi077 ceph-mon[89212]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-05-06T14:05:35.510 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:35 smithi170 ceph-mon[65866]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 170 B/s wr, 124 op/s 2024-05-06T14:05:35.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:35 smithi077 ceph-mon[84247]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 170 B/s wr, 124 op/s 2024-05-06T14:05:35.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:35 smithi077 ceph-mon[89212]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 170 B/s wr, 124 op/s 2024-05-06T14:05:36.821 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 14:05:36 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=cleanup t=2024-05-06T14:05:36.4207267Z level=info msg="Completed cleanup jobs" duration=6.467286ms 2024-05-06T14:05:37.797 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:37 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:37.797 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:37 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:37.797 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:37 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:37.798 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:37 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:37.798 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:37 smithi077 ceph-mon[89212]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 167 op/s 2024-05-06T14:05:37.798 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:37 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:37.798 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:37 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:37.798 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:37 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:37.798 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:37 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:37.798 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:37 smithi077 ceph-mon[84247]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 167 op/s 2024-05-06T14:05:37.821 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:37 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:37.821 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:37 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:37.821 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:37 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:37.821 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:37 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:37.821 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:37 smithi170 ceph-mon[65866]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 167 op/s 2024-05-06T14:05:38.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:38 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:38.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:38 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:38.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:38 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:38.527 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:38 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:38.797 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:38 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:38.798 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:38 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:38.798 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:38 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:38.798 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:38 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:38.798 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:38 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:38.798 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:38 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:38.799 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:38 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:38.799 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:38 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:39.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:39 smithi170 ceph-mon[65866]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-06T14:05:39.798 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:39 smithi077 ceph-mon[84247]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-06T14:05:39.798 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:39 smithi077 ceph-mon[89212]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-05-06T14:05:42.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:05:42.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:05:42.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:05:42.049 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 85 B/s wr, 140 op/s 2024-05-06T14:05:42.049 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:05:42.049 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.049 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.049 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.049 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.049 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.049 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.049 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all rgw 2024-05-06T14:05:42.049 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.049 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi077.xctpgy"}]: dispatch 2024-05-06T14:05:42.049 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi077.xctpgy"}]': finished 2024-05-06T14:05:42.049 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi170.rackmk"}]: dispatch 2024-05-06T14:05:42.049 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi170.rackmk"}]': finished 2024-05-06T14:05:42.049 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.050 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.050 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.050 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.050 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.050 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.050 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.050 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.050 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.050 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.050 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.050 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.050 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.050 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.050 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.050 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-05-06T14:05:42.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-05-06T14:05:42.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:05:42.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-05-06T14:05:42.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-05-06T14:05:42.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-05-06T14:05:42.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-05-06T14:05:42.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-05-06T14:05:42.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-05-06T14:05:42.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-05-06T14:05:42.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-05-06T14:05:42.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-05-06T14:05:42.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-05-06T14:05:42.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-05-06T14:05:42.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:05:42.051 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-05-06T14:05:42.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:05:42.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:05:42.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:05:42.052 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 85 B/s wr, 140 op/s 2024-05-06T14:05:42.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:05:42.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all rgw 2024-05-06T14:05:42.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi077.xctpgy"}]: dispatch 2024-05-06T14:05:42.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi077.xctpgy"}]': finished 2024-05-06T14:05:42.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi170.rackmk"}]: dispatch 2024-05-06T14:05:42.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi170.rackmk"}]': finished 2024-05-06T14:05:42.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.053 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.054 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-05-06T14:05:42.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-05-06T14:05:42.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-05-06T14:05:42.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-05-06T14:05:42.054 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-05-06T14:05:42.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-05-06T14:05:42.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:05:42.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:05:42.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:05:42.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:05:42.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:05:42.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:05:42.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-05-06T14:05:42.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-05-06T14:05:42.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:05:42.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:05:42.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.055 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:05:42.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:05:42.056 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.056 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-05-06T14:05:42.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-05-06T14:05:42.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:05:42.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-05-06T14:05:42.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-05-06T14:05:42.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-05-06T14:05:42.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-05-06T14:05:42.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-05-06T14:05:42.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-05-06T14:05:42.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-05-06T14:05:42.057 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-05-06T14:05:42.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-05-06T14:05:42.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-05-06T14:05:42.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-05-06T14:05:42.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:05:42.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-05-06T14:05:42.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-05-06T14:05:42.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-05-06T14:05:42.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-05-06T14:05:42.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-05-06T14:05:42.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-05-06T14:05:42.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-05-06T14:05:42.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:05:42.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:05:42.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:05:42.058 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:05:42.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:05:42.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:05:42.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-05-06T14:05:42.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-05-06T14:05:42.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:05:42.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:05:42.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:05:42.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:05:42.059 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.071 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:05:42.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:05:42.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:05:42.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 85 B/s wr, 140 op/s 2024-05-06T14:05:42.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:05:42.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.073 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.073 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all rgw 2024-05-06T14:05:42.073 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.073 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi077.xctpgy"}]: dispatch 2024-05-06T14:05:42.073 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi077.xctpgy"}]': finished 2024-05-06T14:05:42.073 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi170.rackmk"}]: dispatch 2024-05-06T14:05:42.073 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi170.rackmk"}]': finished 2024-05-06T14:05:42.073 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.073 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.073 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.073 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.073 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.073 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.073 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.074 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.074 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.074 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.074 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.074 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.074 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.074 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.074 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:05:42.074 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-05-06T14:05:42.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-05-06T14:05:42.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:05:42.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-05-06T14:05:42.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-05-06T14:05:42.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-05-06T14:05:42.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-05-06T14:05:42.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-05-06T14:05:42.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-05-06T14:05:42.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-05-06T14:05:42.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-05-06T14:05:42.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-05-06T14:05:42.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-05-06T14:05:42.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-05-06T14:05:42.075 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:05:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-05-06T14:05:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-05-06T14:05:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-05-06T14:05:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-05-06T14:05:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-05-06T14:05:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-05-06T14:05:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-05-06T14:05:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:05:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:05:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:05:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:05:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:05:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-05-06T14:05:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-05-06T14:05:42.076 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-05-06T14:05:42.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:05:42.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:05:42.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:42.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:05:42.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:05:42.077 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:05:43.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:43 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all rbd-mirror 2024-05-06T14:05:43.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:43 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all ceph-exporter 2024-05-06T14:05:43.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:43 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all nfs 2024-05-06T14:05:43.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:43 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all nvmeof 2024-05-06T14:05:43.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:43 smithi077 ceph-mon[89212]: Upgrade: Finalizing container_image settings 2024-05-06T14:05:43.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:43 smithi077 ceph-mon[89212]: Upgrade: Complete! 2024-05-06T14:05:43.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:43 smithi077 ceph-mon[89212]: Checking dashboard <-> RGW credentials 2024-05-06T14:05:43.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:43 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all rbd-mirror 2024-05-06T14:05:43.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:43 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all ceph-exporter 2024-05-06T14:05:43.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:43 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all nfs 2024-05-06T14:05:43.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:43 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all nvmeof 2024-05-06T14:05:43.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:43 smithi077 ceph-mon[84247]: Upgrade: Finalizing container_image settings 2024-05-06T14:05:43.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:43 smithi077 ceph-mon[84247]: Upgrade: Complete! 2024-05-06T14:05:43.549 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:43 smithi077 ceph-mon[84247]: Checking dashboard <-> RGW credentials 2024-05-06T14:05:43.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:43 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all rbd-mirror 2024-05-06T14:05:43.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:43 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all ceph-exporter 2024-05-06T14:05:43.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:43 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all nfs 2024-05-06T14:05:43.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:43 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all nvmeof 2024-05-06T14:05:43.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:43 smithi170 ceph-mon[65866]: Upgrade: Finalizing container_image settings 2024-05-06T14:05:43.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:43 smithi170 ceph-mon[65866]: Upgrade: Complete! 2024-05-06T14:05:43.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:43 smithi170 ceph-mon[65866]: Checking dashboard <-> RGW credentials 2024-05-06T14:05:44.047 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:05:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:05:43] "GET /metrics HTTP/1.1" 200 38411 "" "Prometheus/2.43.0" 2024-05-06T14:05:44.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:44 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:05:44.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:44 smithi077 ceph-mon[84247]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 85 B/s wr, 128 op/s 2024-05-06T14:05:44.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:44 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:05:44.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:44 smithi077 ceph-mon[89212]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 85 B/s wr, 128 op/s 2024-05-06T14:05:44.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:44 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:05:44.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:44 smithi170 ceph-mon[65866]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 85 B/s wr, 128 op/s 2024-05-06T14:05:45.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:45 smithi077 ceph-mon[84247]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 85 B/s wr, 86 op/s 2024-05-06T14:05:45.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:45 smithi077 ceph-mon[89212]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 85 B/s wr, 86 op/s 2024-05-06T14:05:45.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:45 smithi170 ceph-mon[65866]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 85 B/s wr, 86 op/s 2024-05-06T14:05:47.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:47 smithi077 ceph-mon[84247]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 46 op/s 2024-05-06T14:05:47.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:47 smithi077 ceph-mon[89212]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 46 op/s 2024-05-06T14:05:47.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:47 smithi170 ceph-mon[65866]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 46 op/s 2024-05-06T14:05:49.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:49 smithi077 ceph-mon[84247]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 3 op/s 2024-05-06T14:05:49.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:49 smithi077 ceph-mon[89212]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 3 op/s 2024-05-06T14:05:49.572 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:49 smithi170 ceph-mon[65866]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 3 op/s 2024-05-06T14:05:51.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:51 smithi077 ceph-mon[84247]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-05-06T14:05:51.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:51 smithi077 ceph-mon[89212]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-05-06T14:05:51.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:51 smithi170 ceph-mon[65866]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-05-06T14:05:53.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:53 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:05:53.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:53 smithi077 ceph-mon[84247]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-05-06T14:05:53.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:53 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:05:53.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:53 smithi077 ceph-mon[89212]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-05-06T14:05:53.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:53 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:05:53.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:53 smithi170 ceph-mon[65866]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-05-06T14:05:54.047 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:05:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:05:53] "GET /metrics HTTP/1.1" 200 38411 "" "Prometheus/2.43.0" 2024-05-06T14:05:54.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:54 smithi077 ceph-mon[84247]: from='client.64729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:54.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:54 smithi077 ceph-mon[89212]: from='client.64729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:54.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:54 smithi170 ceph-mon[65866]: from='client.64729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:54.619 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch ps' 2024-05-06T14:05:55.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:55 smithi077 ceph-mon[84247]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-05-06T14:05:55.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:55 smithi077 ceph-mon[89212]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-05-06T14:05:55.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:55 smithi170 ceph-mon[65866]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-05-06T14:05:56.509 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T14:05:56.509 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (10m) 20s ago 33m 26.0M - 0.25.0 c8568f914cd2 997175e3e64e 2024-05-06T14:05:56.509 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (10m) 19s ago 32m 81.2M - 9.4.12 2bacad6d85d8 140bb9a2760a 2024-05-06T14:05:56.509 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (11m) 20s ago 32m 64.4M - 3.5 e1d6a67b021e 9353a303fb55 2024-05-06T14:05:56.509 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283,8765 running (12m) 19s ago 37m 453M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 125521be53a3 2024-05-06T14:05:56.509 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (21m) 20s ago 39m 514M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T14:05:56.509 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (8m) 20s ago 39m 58.2M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 5e50731ccbe8 2024-05-06T14:05:56.509 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (9m) 19s ago 37m 49.9M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 61cd48b3d0ff 2024-05-06T14:05:56.509 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (8m) 20s ago 38m 47.6M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 8a473cc2ffd1 2024-05-06T14:05:56.510 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (11m) 20s ago 33m 16.4M - 1.5.0 0da6a335fe13 67f45fa4968b 2024-05-06T14:05:56.510 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (11m) 19s ago 33m 15.9M - 1.5.0 0da6a335fe13 02081b71a5dd 2024-05-06T14:05:56.510 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (6m) 20s ago 37m 52.6M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 fbe876a933ed 2024-05-06T14:05:56.510 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (5m) 20s ago 36m 54.0M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 6f664e567c93 2024-05-06T14:05:56.510 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (4m) 20s ago 36m 51.5M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 a4c3266b87f8 2024-05-06T14:05:56.510 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (3m) 20s ago 36m 52.5M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 7c9e6dbacb88 2024-05-06T14:05:56.510 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (3m) 19s ago 35m 51.3M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 345a6ce2c7c3 2024-05-06T14:05:56.510 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (2m) 19s ago 35m 49.1M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 fa8ea71417a6 2024-05-06T14:05:56.510 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (2m) 19s ago 34m 49.6M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 1fbae25547b9 2024-05-06T14:05:56.510 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (98s) 19s ago 34m 55.0M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 82e3dd2bbdd9 2024-05-06T14:05:56.510 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (11m) 19s ago 33m 73.0M - 2.43.0 a07b618ecd1d 10d5c513bb9b 2024-05-06T14:05:56.510 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (29s) 20s ago 32m 74.2M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 bfc48a115dcd 2024-05-06T14:05:56.510 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (24s) 19s ago 32m 76.1M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 d5e5069aa8a4 2024-05-06T14:05:56.513 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:56 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:05:56.513 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:56 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:05:56.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:56 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:05:57.244 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph versions | jq -e '"'"'.rgw | length == 1'"'"'' 2024-05-06T14:05:57.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:57 smithi077 ceph-mon[84247]: from='client.54801 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:57.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:57 smithi077 ceph-mon[84247]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:05:57.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:57 smithi077 ceph-mon[89212]: from='client.54801 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:57.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:57 smithi077 ceph-mon[89212]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:05:57.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:57 smithi170 ceph-mon[65866]: from='client.54801 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:05:57.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:57 smithi170 ceph-mon[65866]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:05:59.373 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T14:05:59.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:05:59 smithi077 ceph-mon[84247]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:05:59.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:05:59 smithi077 ceph-mon[89212]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:05:59.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:05:59 smithi170 ceph-mon[65866]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:06:00.271 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph versions | jq -e '"'"'.rgw | keys'"'"' | grep $sha1' 2024-05-06T14:06:00.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:00 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/849483730' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:00.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:00 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/849483730' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:00.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:00 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/849483730' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:01.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:01 smithi077 ceph-mon[89212]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:06:01.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:01 smithi077 ceph-mon[84247]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:06:01.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:01 smithi170 ceph-mon[65866]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:06:02.373 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)" 2024-05-06T14:06:02.705 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:02 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/177203062' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:02.705 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:02 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/177203062' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:02.821 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:02 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/177203062' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:03.018 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch upgrade status' 2024-05-06T14:06:03.579 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:03 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:06:03.579 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:03 smithi077 ceph-mon[84247]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:06:03.579 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:03 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:06:03.579 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:03 smithi077 ceph-mon[89212]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:06:03.821 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:03 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:06:03.821 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:03 smithi170 ceph-mon[65866]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:06:04.047 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:06:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:06:03] "GET /metrics HTTP/1.1" 200 38409 "" "Prometheus/2.43.0" 2024-05-06T14:06:05.185 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:06:05.186 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": null, 2024-05-06T14:06:05.186 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": false, 2024-05-06T14:06:05.186 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "", 2024-05-06T14:06:05.186 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [], 2024-05-06T14:06:05.186 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": null, 2024-05-06T14:06:05.186 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "", 2024-05-06T14:06:05.186 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-05-06T14:06:05.186 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:06:05.536 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:05 smithi077 ceph-mon[84247]: from='client.54819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:06:05.536 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:05 smithi077 ceph-mon[84247]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:06:05.536 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:05 smithi077 ceph-mon[89212]: from='client.54819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:06:05.537 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:05 smithi077 ceph-mon[89212]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:06:05.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:05 smithi170 ceph-mon[65866]: from='client.54819 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:06:05.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:05 smithi170 ceph-mon[65866]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:06:05.877 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph health detail' 2024-05-06T14:06:07.403 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:07 smithi077 ceph-mon[84247]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:06:07.403 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:07 smithi077 ceph-mon[89212]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:06:07.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:07 smithi170 ceph-mon[65866]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:06:07.909 INFO:teuthology.orchestra.run.smithi077.stdout:HEALTH_OK 2024-05-06T14:06:08.478 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:08 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/2370110343' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T14:06:08.479 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:08 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/2370110343' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T14:06:08.503 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-05-06T14:06:08.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:08 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/2370110343' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T14:06:09.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:09 smithi077 ceph-mon[84247]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:06:09.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:09 smithi077 ceph-mon[89212]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:06:09.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:09 smithi170 ceph-mon[65866]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:06:10.429 INFO:teuthology.orchestra.run.smithi077.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:06:11.242 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-06T14:06:11.258 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi077.front.sepia.ceph.com 2024-05-06T14:06:11.258 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-05-06T14:06:11.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:11 smithi077 ceph-mon[84247]: from='client.64765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:06:11.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:11 smithi077 ceph-mon[84247]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:06:11.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:11 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:11.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:11 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:06:11.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:11 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:06:11.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:11 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:11.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:11 smithi077 ceph-mon[84247]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:06:11.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:11 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:06:11.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:11 smithi077 ceph-mon[84247]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:06:11.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:11 smithi077 ceph-mon[89212]: from='client.64765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:06:11.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:11 smithi077 ceph-mon[89212]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:06:11.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:11 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:11.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:11 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:06:11.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:11 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:06:11.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:11 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:11.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:11 smithi077 ceph-mon[89212]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:06:11.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:11 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:06:11.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:11 smithi077 ceph-mon[89212]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:06:11.821 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:11 smithi170 ceph-mon[65866]: from='client.64765 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:06:11.821 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:11 smithi170 ceph-mon[65866]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:06:11.821 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:11 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:11.821 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:11 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:06:11.821 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:11 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:06:11.821 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:11 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:11.822 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:11 smithi170 ceph-mon[65866]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:06:11.822 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:11 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:06:11.822 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:11 smithi170 ceph-mon[65866]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:06:13.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:13 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:06:13.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:13 smithi077 ceph-mon[84247]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:06:13.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:13 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:06:13.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:13 smithi077 ceph-mon[89212]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:06:13.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:13 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:06:13.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:13 smithi170 ceph-mon[65866]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:06:13.990 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:06:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:06:13] "GET /metrics HTTP/1.1" 200 38411 "" "Prometheus/2.43.0" 2024-05-06T14:06:14.256 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T14:06:14.934 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T14:06:14.934 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (11m) 38s ago 33m 26.0M - 0.25.0 c8568f914cd2 997175e3e64e 2024-05-06T14:06:14.935 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (10m) 38s ago 33m 81.2M - 9.4.12 2bacad6d85d8 140bb9a2760a 2024-05-06T14:06:14.935 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (11m) 38s ago 32m 64.4M - 3.5 e1d6a67b021e 9353a303fb55 2024-05-06T14:06:14.935 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283,8765 running (12m) 38s ago 37m 453M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 125521be53a3 2024-05-06T14:06:14.935 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (22m) 38s ago 39m 514M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T14:06:14.935 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (8m) 38s ago 39m 58.2M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 5e50731ccbe8 2024-05-06T14:06:14.935 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (9m) 38s ago 38m 49.9M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 61cd48b3d0ff 2024-05-06T14:06:14.935 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (8m) 38s ago 38m 47.6M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 8a473cc2ffd1 2024-05-06T14:06:14.935 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (12m) 38s ago 34m 16.4M - 1.5.0 0da6a335fe13 67f45fa4968b 2024-05-06T14:06:14.935 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (11m) 38s ago 34m 15.9M - 1.5.0 0da6a335fe13 02081b71a5dd 2024-05-06T14:06:14.935 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (6m) 38s ago 37m 52.6M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 fbe876a933ed 2024-05-06T14:06:14.935 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (6m) 38s ago 37m 54.0M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 6f664e567c93 2024-05-06T14:06:14.935 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (5m) 38s ago 36m 51.5M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 a4c3266b87f8 2024-05-06T14:06:14.935 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (4m) 38s ago 36m 52.5M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 7c9e6dbacb88 2024-05-06T14:06:14.935 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (3m) 38s ago 36m 51.3M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 345a6ce2c7c3 2024-05-06T14:06:14.936 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (3m) 38s ago 35m 49.1M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 fa8ea71417a6 2024-05-06T14:06:14.936 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (2m) 38s ago 35m 49.6M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 1fbae25547b9 2024-05-06T14:06:14.936 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (117s) 38s ago 34m 55.0M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 82e3dd2bbdd9 2024-05-06T14:06:14.936 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (11m) 38s ago 34m 73.0M - 2.43.0 a07b618ecd1d 10d5c513bb9b 2024-05-06T14:06:14.936 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (48s) 38s ago 32m 74.2M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 bfc48a115dcd 2024-05-06T14:06:14.936 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (42s) 38s ago 32m 76.1M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 d5e5069aa8a4 2024-05-06T14:06:15.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:15.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (squid) 2024-05-06T14:06:15.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T14:06:15.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:06:15.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:15.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all mgr 2024-05-06T14:06:15.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:15.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:15.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all mon 2024-05-06T14:06:15.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:15.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:15.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all crash 2024-05-06T14:06:15.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:15.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:15.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all osd 2024-05-06T14:06:15.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:15.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:15.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all mds 2024-05-06T14:06:15.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:15.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:15.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all rgw 2024-05-06T14:06:15.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:15.048 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:15.049 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all rbd-mirror 2024-05-06T14:06:15.049 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:15.049 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:15.049 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all cephfs-mirror 2024-05-06T14:06:15.049 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:15.049 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: Upgrade: Setting container_image for all ceph-exporter 2024-05-06T14:06:15.049 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:15.049 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:15.049 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (squid) 2024-05-06T14:06:15.049 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T14:06:15.049 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:06:15.049 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:15.049 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all mgr 2024-05-06T14:06:15.050 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:15.050 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:15.050 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all mon 2024-05-06T14:06:15.050 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:15.050 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:15.050 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all crash 2024-05-06T14:06:15.050 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:15.050 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:15.050 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all osd 2024-05-06T14:06:15.050 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:15.050 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:15.050 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all mds 2024-05-06T14:06:15.050 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:15.050 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:15.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all rgw 2024-05-06T14:06:15.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:15.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:15.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all rbd-mirror 2024-05-06T14:06:15.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:15.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:15.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all cephfs-mirror 2024-05-06T14:06:15.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:15.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: Upgrade: Setting container_image for all ceph-exporter 2024-05-06T14:06:15.051 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:14 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:15.071 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:15.071 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: Upgrade: Target is version 19.0.0-2498-g70ce9539 (squid) 2024-05-06T14:06:15.071 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0150ca77ab7330820ade0517ccbade81a6c60dbd326dfc0e8f481c3af7950ccb'] 2024-05-06T14:06:15.071 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-06T14:06:15.071 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:15.071 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all mgr 2024-05-06T14:06:15.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:15.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:15.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all mon 2024-05-06T14:06:15.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:15.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:15.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all crash 2024-05-06T14:06:15.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:15.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:15.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all osd 2024-05-06T14:06:15.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:15.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:15.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all mds 2024-05-06T14:06:15.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:15.072 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:15.073 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all rgw 2024-05-06T14:06:15.073 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:15.073 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:15.073 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all rbd-mirror 2024-05-06T14:06:15.073 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:15.073 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:15.073 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all cephfs-mirror 2024-05-06T14:06:15.073 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:15.073 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: Upgrade: Setting container_image for all ceph-exporter 2024-05-06T14:06:15.073 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:14 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:15.339 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:06:15.339 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-05-06T14:06:15.340 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T14:06:15.340 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:06:15.340 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-05-06T14:06:15.340 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T14:06:15.340 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:06:15.340 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-05-06T14:06:15.340 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 8 2024-05-06T14:06:15.340 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:06:15.340 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw": { 2024-05-06T14:06:15.340 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T14:06:15.340 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:06:15.340 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-05-06T14:06:15.340 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 15 2024-05-06T14:06:15.340 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T14:06:15.341 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:06:15.687 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:06:15.688 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T14:06:15.688 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-05-06T14:06:15.688 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-06T14:06:15.688 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [ 2024-05-06T14:06:15.688 INFO:teuthology.orchestra.run.smithi077.stdout: "mon", 2024-05-06T14:06:15.688 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw", 2024-05-06T14:06:15.688 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr", 2024-05-06T14:06:15.688 INFO:teuthology.orchestra.run.smithi077.stdout: "osd" 2024-05-06T14:06:15.688 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-05-06T14:06:15.688 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "15/21 daemons upgraded", 2024-05-06T14:06:15.688 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "Currently upgrading iscsi daemons", 2024-05-06T14:06:15.688 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-05-06T14:06:15.688 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:06:16.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:15 smithi077 ceph-mon[89212]: from='client.54837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:06:16.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:15 smithi077 ceph-mon[89212]: from='client.64777 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:06:16.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:15 smithi077 ceph-mon[89212]: from='client.54849 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:06:16.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:15 smithi077 ceph-mon[89212]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:06:16.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:15 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/4264255408' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:16.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:15 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:16.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:15 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T14:06:16.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:15 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:06:16.048 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:15 smithi077 ceph-mon[84247]: from='client.54837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:06:16.048 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:15 smithi077 ceph-mon[84247]: from='client.64777 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:06:16.048 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:15 smithi077 ceph-mon[84247]: from='client.54849 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:06:16.048 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:15 smithi077 ceph-mon[84247]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:06:16.048 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:15 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/4264255408' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:16.048 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:15 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:16.048 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:15 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T14:06:16.048 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:15 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:06:16.071 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:15 smithi170 ceph-mon[65866]: from='client.54837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:06:16.071 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:15 smithi170 ceph-mon[65866]: from='client.64777 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:06:16.071 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:15 smithi170 ceph-mon[65866]: from='client.54849 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:06:16.071 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:15 smithi170 ceph-mon[65866]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:06:16.071 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:15 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/4264255408' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:16.071 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:15 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:16.071 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:15 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi077.muddwz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-06T14:06:16.071 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:15 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:06:16.101 INFO:teuthology.orchestra.run.smithi077.stdout:HEALTH_OK 2024-05-06T14:06:17.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:16 smithi077 ceph-mon[89212]: Upgrade: Updating iscsi.foo.smithi077.muddwz 2024-05-06T14:06:17.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:16 smithi077 ceph-mon[89212]: Deploying daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T14:06:17.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:16 smithi077 ceph-mon[89212]: from='client.64795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:06:17.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:16 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/730612540' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T14:06:17.048 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:16 smithi077 ceph-mon[84247]: Upgrade: Updating iscsi.foo.smithi077.muddwz 2024-05-06T14:06:17.048 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:16 smithi077 ceph-mon[84247]: Deploying daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T14:06:17.048 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:16 smithi077 ceph-mon[84247]: from='client.64795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:06:17.048 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:16 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/730612540' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T14:06:17.071 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:16 smithi170 ceph-mon[65866]: Upgrade: Updating iscsi.foo.smithi077.muddwz 2024-05-06T14:06:17.071 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:16 smithi170 ceph-mon[65866]: Deploying daemon iscsi.foo.smithi077.muddwz on smithi077 2024-05-06T14:06:17.071 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:16 smithi170 ceph-mon[65866]: from='client.64795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:06:17.071 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:16 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/730612540' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T14:06:18.047 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:17 smithi077 ceph-mon[89212]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:06:18.048 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:17 smithi077 ceph-mon[84247]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:06:18.071 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:17 smithi170 ceph-mon[65866]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:06:19.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:19 smithi077 ceph-mon[84247]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:06:19.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:19 smithi077 ceph-mon[89212]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:06:19.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:19 smithi170 ceph-mon[65866]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:06:21.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:21 smithi077 ceph-mon[84247]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:06:21.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:21 smithi077 ceph-mon[89212]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:06:21.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:21 smithi170 ceph-mon[65866]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:06:23.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:23 smithi077 ceph-mon[84247]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:06:23.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:23 smithi077 ceph-mon[84247]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:06:23.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:23 smithi077 ceph-mon[89212]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:06:23.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:23 smithi077 ceph-mon[89212]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:06:23.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:23 smithi170 ceph-mon[65866]: from='client.15684 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:06:23.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:23 smithi170 ceph-mon[65866]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:06:24.047 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:06:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:06:23] "GET /metrics HTTP/1.1" 200 38411 "" "Prometheus/2.43.0" 2024-05-06T14:06:25.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:25 smithi077 ceph-mon[84247]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:06:25.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:25 smithi077 ceph-mon[89212]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:06:25.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:25 smithi170 ceph-mon[65866]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:06:26.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:26 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:06:26.797 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:26 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:06:26.798 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:26 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:06:27.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:27 smithi170 ceph-mon[65866]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:06:27.774 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:27 smithi077 ceph-mon[84247]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:06:27.774 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:27 smithi077 ceph-mon[89212]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:06:29.529 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:06:29 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T14:06:29.156+0000 7f5c9abd8700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi077.muddwz on host smithi077 failed. 2024-05-06T14:06:30.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:30 smithi077 ceph-mon[84247]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi077.muddwz on host smithi077 failed. 2024-05-06T14:06:30.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:30 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:30.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:30 smithi077 ceph-mon[84247]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-06T14:06:30.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:30 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:06:30.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:30 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:06:30.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:30 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:30.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:30 smithi077 ceph-mon[89212]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi077.muddwz on host smithi077 failed. 2024-05-06T14:06:30.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:30 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:30.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:30 smithi077 ceph-mon[89212]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-06T14:06:30.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:30 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:06:30.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:30 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:06:30.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:30 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:30.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:30 smithi170 ceph-mon[65866]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi077.muddwz on host smithi077 failed. 2024-05-06T14:06:30.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:30 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:30.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:30 smithi170 ceph-mon[65866]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-06T14:06:30.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:30 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:06:30.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:30 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:06:30.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:30 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:31.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:31 smithi077 ceph-mon[84247]: Health check failed: Upgrading daemon iscsi.foo.smithi077.muddwz on host smithi077 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-05-06T14:06:31.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:31 smithi077 ceph-mon[89212]: Health check failed: Upgrading daemon iscsi.foo.smithi077.muddwz on host smithi077 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-05-06T14:06:31.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:31 smithi170 ceph-mon[65866]: Health check failed: Upgrading daemon iscsi.foo.smithi077.muddwz on host smithi077 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-05-06T14:06:32.447 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:32 smithi077 ceph-mon[89212]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-05-06T14:06:32.447 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:32 smithi077 ceph-mon[84247]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-05-06T14:06:32.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:32 smithi170 ceph-mon[65866]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-05-06T14:06:34.047 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:06:33 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:06:33] "GET /metrics HTTP/1.1" 200 38408 "" "Prometheus/2.43.0" 2024-05-06T14:06:34.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:34 smithi077 ceph-mon[84247]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-05-06T14:06:34.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:34 smithi077 ceph-mon[89212]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-05-06T14:06:34.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:34 smithi170 ceph-mon[65866]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-05-06T14:06:36.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:36 smithi077 ceph-mon[84247]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-05-06T14:06:36.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:36 smithi077 ceph-mon[89212]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-05-06T14:06:36.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:36 smithi170 ceph-mon[65866]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-05-06T14:06:38.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:38 smithi077 ceph-mon[84247]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 428 B/s rd, 0 op/s 2024-05-06T14:06:38.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:38 smithi077 ceph-mon[89212]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 428 B/s rd, 0 op/s 2024-05-06T14:06:38.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:38 smithi170 ceph-mon[65866]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 428 B/s rd, 0 op/s 2024-05-06T14:06:40.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:40 smithi077 ceph-mon[84247]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 428 B/s rd, 0 op/s 2024-05-06T14:06:40.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:40 smithi077 ceph-mon[89212]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 428 B/s rd, 0 op/s 2024-05-06T14:06:40.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:40 smithi170 ceph-mon[65866]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 428 B/s rd, 0 op/s 2024-05-06T14:06:41.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:41 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/1761426576' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T14:06:41.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:41 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/2639603715' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/1002982425"}]: dispatch 2024-05-06T14:06:41.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:41.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:06:41.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:41 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/1761426576' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T14:06:41.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:41 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/2639603715' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/1002982425"}]: dispatch 2024-05-06T14:06:41.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:41.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:06:41.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:41 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/1761426576' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-06T14:06:41.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:41 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/2639603715' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/1002982425"}]: dispatch 2024-05-06T14:06:41.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:06:41.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:06:42.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:42 smithi077 ceph-mon[84247]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail 2024-05-06T14:06:42.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:42 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/2639603715' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/1002982425"}]': finished 2024-05-06T14:06:42.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:42 smithi077 ceph-mon[84247]: osdmap e155: 8 total, 8 up, 8 in 2024-05-06T14:06:42.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:42 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/2224251244' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2932646164"}]: dispatch 2024-05-06T14:06:42.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:42 smithi077 ceph-mon[84247]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2932646164"}]: dispatch 2024-05-06T14:06:42.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:42 smithi077 ceph-mon[89212]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail 2024-05-06T14:06:42.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:42 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/2639603715' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/1002982425"}]': finished 2024-05-06T14:06:42.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:42 smithi077 ceph-mon[89212]: osdmap e155: 8 total, 8 up, 8 in 2024-05-06T14:06:42.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:42 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/2224251244' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2932646164"}]: dispatch 2024-05-06T14:06:42.548 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:42 smithi077 ceph-mon[89212]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2932646164"}]: dispatch 2024-05-06T14:06:42.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:42 smithi170 ceph-mon[65866]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail 2024-05-06T14:06:42.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:42 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/2639603715' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6801/1002982425"}]': finished 2024-05-06T14:06:42.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:42 smithi170 ceph-mon[65866]: osdmap e155: 8 total, 8 up, 8 in 2024-05-06T14:06:42.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:42 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/2224251244' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2932646164"}]: dispatch 2024-05-06T14:06:42.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:42 smithi170 ceph-mon[65866]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2932646164"}]: dispatch 2024-05-06T14:06:43.472 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:43 smithi170 ceph-mon[65866]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2932646164"}]': finished 2024-05-06T14:06:43.472 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:43 smithi170 ceph-mon[65866]: osdmap e156: 8 total, 8 up, 8 in 2024-05-06T14:06:43.472 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:43 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/3527777269' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/1002982425"}]: dispatch 2024-05-06T14:06:43.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:43 smithi077 ceph-mon[84247]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2932646164"}]': finished 2024-05-06T14:06:43.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:43 smithi077 ceph-mon[84247]: osdmap e156: 8 total, 8 up, 8 in 2024-05-06T14:06:43.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:43 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/3527777269' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/1002982425"}]: dispatch 2024-05-06T14:06:43.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:43 smithi077 ceph-mon[89212]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2932646164"}]': finished 2024-05-06T14:06:43.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:43 smithi077 ceph-mon[89212]: osdmap e156: 8 total, 8 up, 8 in 2024-05-06T14:06:43.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:43 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/3527777269' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/1002982425"}]: dispatch 2024-05-06T14:06:44.046 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:06:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:06:43] "GET /metrics HTTP/1.1" 200 38487 "" "Prometheus/2.43.0" 2024-05-06T14:06:44.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:44 smithi077 ceph-mon[89212]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail 2024-05-06T14:06:44.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:44 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/3527777269' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/1002982425"}]': finished 2024-05-06T14:06:44.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:44 smithi077 ceph-mon[89212]: osdmap e157: 8 total, 8 up, 8 in 2024-05-06T14:06:44.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:44 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/873232941' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/300671891"}]: dispatch 2024-05-06T14:06:44.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:44 smithi077 ceph-mon[89212]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/300671891"}]: dispatch 2024-05-06T14:06:44.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:44 smithi077 ceph-mon[84247]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail 2024-05-06T14:06:44.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:44 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/3527777269' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/1002982425"}]': finished 2024-05-06T14:06:44.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:44 smithi077 ceph-mon[84247]: osdmap e157: 8 total, 8 up, 8 in 2024-05-06T14:06:44.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:44 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/873232941' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/300671891"}]: dispatch 2024-05-06T14:06:44.548 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:44 smithi077 ceph-mon[84247]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/300671891"}]: dispatch 2024-05-06T14:06:44.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:44 smithi170 ceph-mon[65866]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail 2024-05-06T14:06:44.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:44 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/3527777269' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:6800/1002982425"}]': finished 2024-05-06T14:06:44.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:44 smithi170 ceph-mon[65866]: osdmap e157: 8 total, 8 up, 8 in 2024-05-06T14:06:44.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:44 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/873232941' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/300671891"}]: dispatch 2024-05-06T14:06:44.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:44 smithi170 ceph-mon[65866]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/300671891"}]: dispatch 2024-05-06T14:06:45.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:45 smithi077 ceph-mon[84247]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/300671891"}]': finished 2024-05-06T14:06:45.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:45 smithi077 ceph-mon[84247]: osdmap e158: 8 total, 8 up, 8 in 2024-05-06T14:06:45.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:45 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/182645814' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2845246402"}]: dispatch 2024-05-06T14:06:45.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:45 smithi077 ceph-mon[84247]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2845246402"}]: dispatch 2024-05-06T14:06:45.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:45 smithi077 ceph-mon[89212]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/300671891"}]': finished 2024-05-06T14:06:45.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:45 smithi077 ceph-mon[89212]: osdmap e158: 8 total, 8 up, 8 in 2024-05-06T14:06:45.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:45 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/182645814' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2845246402"}]: dispatch 2024-05-06T14:06:45.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:45 smithi077 ceph-mon[89212]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2845246402"}]: dispatch 2024-05-06T14:06:45.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:45 smithi170 ceph-mon[65866]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/300671891"}]': finished 2024-05-06T14:06:45.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:45 smithi170 ceph-mon[65866]: osdmap e158: 8 total, 8 up, 8 in 2024-05-06T14:06:45.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:45 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/182645814' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2845246402"}]: dispatch 2024-05-06T14:06:45.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:45 smithi170 ceph-mon[65866]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2845246402"}]: dispatch 2024-05-06T14:06:46.456 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T14:06:46.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:46 smithi077 ceph-mon[84247]: pgmap v320: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-05-06T14:06:46.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:46 smithi077 ceph-mon[84247]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2845246402"}]': finished 2024-05-06T14:06:46.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:46 smithi077 ceph-mon[84247]: osdmap e159: 8 total, 8 up, 8 in 2024-05-06T14:06:46.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:46 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/1824774788' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/4260800135"}]: dispatch 2024-05-06T14:06:46.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:46 smithi077 ceph-mon[89212]: pgmap v320: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-05-06T14:06:46.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:46 smithi077 ceph-mon[89212]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2845246402"}]': finished 2024-05-06T14:06:46.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:46 smithi077 ceph-mon[89212]: osdmap e159: 8 total, 8 up, 8 in 2024-05-06T14:06:46.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:46 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/1824774788' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/4260800135"}]: dispatch 2024-05-06T14:06:46.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:46 smithi170 ceph-mon[65866]: pgmap v320: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-05-06T14:06:46.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:46 smithi170 ceph-mon[65866]: from='client.? ' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/2845246402"}]': finished 2024-05-06T14:06:46.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:46 smithi170 ceph-mon[65866]: osdmap e159: 8 total, 8 up, 8 in 2024-05-06T14:06:46.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:46 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/1824774788' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/4260800135"}]: dispatch 2024-05-06T14:06:46.787 INFO:teuthology.orchestra.run.smithi077.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi077.muddwz on host smithi077 failed." 2024-05-06T14:06:47.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:47 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/1824774788' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/4260800135"}]': finished 2024-05-06T14:06:47.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:47 smithi077 ceph-mon[84247]: osdmap e160: 8 total, 8 up, 8 in 2024-05-06T14:06:47.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:47 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/1824774788' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/4260800135"}]': finished 2024-05-06T14:06:47.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:47 smithi077 ceph-mon[89212]: osdmap e160: 8 total, 8 up, 8 in 2024-05-06T14:06:47.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:47 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/1824774788' entity='client.iscsi.foo.smithi077.muddwz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.77:0/4260800135"}]': finished 2024-05-06T14:06:47.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:47 smithi170 ceph-mon[65866]: osdmap e160: 8 total, 8 up, 8 in 2024-05-06T14:06:47.573 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch ps' 2024-05-06T14:06:48.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:48 smithi077 ceph-mon[89212]: from='client.64867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:06:48.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:48 smithi077 ceph-mon[89212]: from='client.64873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:06:48.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:48 smithi077 ceph-mon[89212]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-05-06T14:06:48.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:48 smithi077 ceph-mon[84247]: from='client.64867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:06:48.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:48 smithi077 ceph-mon[84247]: from='client.64873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:06:48.547 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:48 smithi077 ceph-mon[84247]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-05-06T14:06:48.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:48 smithi170 ceph-mon[65866]: from='client.64867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:06:48.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:48 smithi170 ceph-mon[65866]: from='client.64873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:06:48.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:48 smithi170 ceph-mon[65866]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-05-06T14:06:49.628 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T14:06:49.628 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (11m) 73s ago 33m 26.0M - 0.25.0 c8568f914cd2 997175e3e64e 2024-05-06T14:06:49.629 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (11m) 73s ago 33m 81.2M - 9.4.12 2bacad6d85d8 140bb9a2760a 2024-05-06T14:06:49.629 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (12m) 73s ago 33m 64.4M - 3.5 e1d6a67b021e 9353a303fb55 2024-05-06T14:06:49.629 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283,8765 running (12m) 73s ago 38m 453M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 125521be53a3 2024-05-06T14:06:49.629 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (22m) 73s ago 40m 514M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T14:06:49.629 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (9m) 73s ago 40m 58.2M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 5e50731ccbe8 2024-05-06T14:06:49.629 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (10m) 73s ago 38m 49.9M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 61cd48b3d0ff 2024-05-06T14:06:49.629 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (8m) 73s ago 38m 47.6M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 8a473cc2ffd1 2024-05-06T14:06:49.629 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (12m) 73s ago 34m 16.4M - 1.5.0 0da6a335fe13 67f45fa4968b 2024-05-06T14:06:49.629 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (12m) 73s ago 34m 15.9M - 1.5.0 0da6a335fe13 02081b71a5dd 2024-05-06T14:06:49.629 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (7m) 73s ago 38m 52.6M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 fbe876a933ed 2024-05-06T14:06:49.629 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (6m) 73s ago 37m 54.0M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 6f664e567c93 2024-05-06T14:06:49.629 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (5m) 73s ago 37m 51.5M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 a4c3266b87f8 2024-05-06T14:06:49.629 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (4m) 73s ago 37m 52.5M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 7c9e6dbacb88 2024-05-06T14:06:49.629 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (4m) 73s ago 36m 51.3M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 345a6ce2c7c3 2024-05-06T14:06:49.630 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (3m) 73s ago 36m 49.1M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 fa8ea71417a6 2024-05-06T14:06:49.630 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (3m) 73s ago 35m 49.6M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 1fbae25547b9 2024-05-06T14:06:49.630 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (2m) 73s ago 35m 55.0M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 82e3dd2bbdd9 2024-05-06T14:06:49.630 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (11m) 73s ago 34m 73.0M - 2.43.0 a07b618ecd1d 10d5c513bb9b 2024-05-06T14:06:49.630 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (82s) 73s ago 33m 74.2M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 bfc48a115dcd 2024-05-06T14:06:49.630 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (77s) 73s ago 33m 76.1M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 d5e5069aa8a4 2024-05-06T14:06:50.296 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:50 smithi077 ceph-mon[89212]: pgmap v324: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-05-06T14:06:50.297 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:50 smithi077 ceph-mon[84247]: pgmap v324: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-05-06T14:06:50.453 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph versions' 2024-05-06T14:06:50.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:50 smithi170 ceph-mon[65866]: pgmap v324: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-05-06T14:06:51.296 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:51 smithi077 ceph-mon[89212]: from='client.64879 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:06:51.297 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:51 smithi077 ceph-mon[89212]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:06:51.297 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:51 smithi077 ceph-mon[84247]: from='client.64879 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:06:51.297 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:51 smithi077 ceph-mon[84247]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:06:51.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:51 smithi170 ceph-mon[65866]: from='client.64879 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:06:51.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:51 smithi170 ceph-mon[65866]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:06:52.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:52 smithi077 ceph-mon[84247]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 736 B/s rd, 0 op/s 2024-05-06T14:06:52.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:52 smithi077 ceph-mon[89212]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 736 B/s rd, 0 op/s 2024-05-06T14:06:52.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:52 smithi170 ceph-mon[65866]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 736 B/s rd, 0 op/s 2024-05-06T14:06:52.648 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:06:52.648 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-05-06T14:06:52.648 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T14:06:52.648 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:06:52.648 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-05-06T14:06:52.648 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T14:06:52.648 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:06:52.649 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-05-06T14:06:52.649 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 8 2024-05-06T14:06:52.649 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:06:52.649 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw": { 2024-05-06T14:06:52.649 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T14:06:52.649 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:06:52.649 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-05-06T14:06:52.649 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 15 2024-05-06T14:06:52.649 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T14:06:52.649 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:06:53.352 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:53 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/990544967' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:53.352 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:53 smithi077 ceph-mon[84247]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-05-06T14:06:53.352 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:53 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/990544967' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:53.353 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:53 smithi077 ceph-mon[89212]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-05-06T14:06:53.377 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-05-06T14:06:53.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:53 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/990544967' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:06:53.571 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:53 smithi170 ceph-mon[65866]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-05-06T14:06:54.023 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:06:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:06:53] "GET /metrics HTTP/1.1" 200 38487 "" "Prometheus/2.43.0" 2024-05-06T14:06:55.106 INFO:teuthology.orchestra.run.smithi077.stdout:wait for servicemap items w/ changing names to refresh 2024-05-06T14:06:55.871 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'sleep 60' 2024-05-06T14:06:56.505 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:56 smithi077 ceph-mon[84247]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T14:06:56.505 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:56 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:06:56.505 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:56 smithi077 ceph-mon[89212]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T14:06:56.506 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:56 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:06:56.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:56 smithi170 ceph-mon[65866]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-05-06T14:06:56.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:56 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:06:58.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:06:58 smithi077 ceph-mon[84247]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 op/s 2024-05-06T14:06:58.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:06:58 smithi077 ceph-mon[89212]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 op/s 2024-05-06T14:06:58.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:06:58 smithi170 ceph-mon[65866]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 936 B/s rd, 0 op/s 2024-05-06T14:07:00.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:00 smithi077 ceph-mon[84247]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:00.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:00 smithi077 ceph-mon[89212]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:00.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:00 smithi170 ceph-mon[65866]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:01.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:01 smithi077 ceph-mon[84247]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:07:01.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:01 smithi077 ceph-mon[89212]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:07:01.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:01 smithi170 ceph-mon[65866]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:07:02.302 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:02 smithi077 ceph-mon[84247]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:02.302 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:02 smithi077 ceph-mon[89212]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:02.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:02 smithi170 ceph-mon[65866]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:04.046 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:07:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:07:03] "GET /metrics HTTP/1.1" 200 38488 "" "Prometheus/2.43.0" 2024-05-06T14:07:04.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:04 smithi077 ceph-mon[84247]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:04.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:04 smithi077 ceph-mon[89212]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:04.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:04 smithi170 ceph-mon[65866]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:06.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:06 smithi077 ceph-mon[84247]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:06.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:06 smithi077 ceph-mon[89212]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:06.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:06 smithi170 ceph-mon[65866]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:08.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:08 smithi077 ceph-mon[84247]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:08.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:08 smithi077 ceph-mon[89212]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:08.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:08 smithi170 ceph-mon[65866]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:10.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:10 smithi077 ceph-mon[84247]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:10.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:10 smithi077 ceph-mon[89212]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:10.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:10 smithi170 ceph-mon[65866]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:11.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:11 smithi077 ceph-mon[84247]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:07:11.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:11 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:07:11.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:11 smithi077 ceph-mon[89212]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:07:11.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:11 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:07:11.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:11 smithi170 ceph-mon[65866]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:07:11.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:11 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:07:12.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:12 smithi077 ceph-mon[84247]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:12.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:12 smithi077 ceph-mon[89212]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:12.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:12 smithi170 ceph-mon[65866]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:13.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:13 smithi077 ceph-mon[84247]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:13.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:13 smithi077 ceph-mon[89212]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:13.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:13 smithi170 ceph-mon[65866]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:14.046 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:07:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:07:13] "GET /metrics HTTP/1.1" 200 38486 "" "Prometheus/2.43.0" 2024-05-06T14:07:16.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:16 smithi077 ceph-mon[84247]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:16.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:16 smithi077 ceph-mon[89212]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:16.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:16 smithi170 ceph-mon[65866]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:18.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:18 smithi077 ceph-mon[89212]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:18.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:18 smithi077 ceph-mon[84247]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:18.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:18 smithi170 ceph-mon[65866]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:20.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:20 smithi077 ceph-mon[89212]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:20.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:20 smithi077 ceph-mon[84247]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:20.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:20 smithi170 ceph-mon[65866]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:21.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:21 smithi077 ceph-mon[84247]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:07:21.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:21 smithi077 ceph-mon[89212]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:07:21.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:21 smithi170 ceph-mon[65866]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:07:22.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:22 smithi077 ceph-mon[84247]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:22.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:22 smithi077 ceph-mon[89212]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:22.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:22 smithi170 ceph-mon[65866]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:24.046 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:07:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:07:23] "GET /metrics HTTP/1.1" 200 38486 "" "Prometheus/2.43.0" 2024-05-06T14:07:24.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:24 smithi077 ceph-mon[84247]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:24.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:24 smithi077 ceph-mon[89212]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:24.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:24 smithi170 ceph-mon[65866]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:26.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:26 smithi077 ceph-mon[84247]: pgmap v342: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:26.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:26 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:07:26.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:26 smithi077 ceph-mon[89212]: pgmap v342: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:26.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:26 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:07:26.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:26 smithi170 ceph-mon[65866]: pgmap v342: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:26.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:26 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:07:28.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:28 smithi077 ceph-mon[84247]: pgmap v343: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:28.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:28 smithi077 ceph-mon[89212]: pgmap v343: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:28.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:28 smithi170 ceph-mon[65866]: pgmap v343: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:30.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:30 smithi077 ceph-mon[84247]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:30.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:30 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:07:30.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:30 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:07:30.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:30 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:07:30.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:30 smithi077 ceph-mon[89212]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:30.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:30 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:07:30.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:30 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:07:30.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:30 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:07:30.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:30 smithi170 ceph-mon[65866]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:30.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:30 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-06T14:07:30.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:30 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-06T14:07:30.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:30 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' 2024-05-06T14:07:31.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:31 smithi077 ceph-mon[84247]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:07:31.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:31 smithi077 ceph-mon[89212]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:07:31.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:31 smithi170 ceph-mon[65866]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:07:32.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:32 smithi077 ceph-mon[84247]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:32.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:32 smithi077 ceph-mon[89212]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:32.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:32 smithi170 ceph-mon[65866]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:33.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:33 smithi077 ceph-mon[84247]: pgmap v346: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:33.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:33 smithi077 ceph-mon[89212]: pgmap v346: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:33.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:33 smithi170 ceph-mon[65866]: pgmap v346: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:34.046 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:07:33 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:07:33] "GET /metrics HTTP/1.1" 200 38488 "" "Prometheus/2.43.0" 2024-05-06T14:07:36.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:36 smithi077 ceph-mon[84247]: pgmap v347: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:36.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:36 smithi077 ceph-mon[89212]: pgmap v347: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:36.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:36 smithi170 ceph-mon[65866]: pgmap v347: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:38.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:38 smithi077 ceph-mon[84247]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:38.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:38 smithi077 ceph-mon[89212]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:38.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:38 smithi170 ceph-mon[65866]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:40.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:40 smithi077 ceph-mon[84247]: pgmap v349: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:40.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:40 smithi077 ceph-mon[89212]: pgmap v349: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:40.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:40 smithi170 ceph-mon[65866]: pgmap v349: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:41.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:41 smithi077 ceph-mon[84247]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:07:41.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:41 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:07:41.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:41 smithi077 ceph-mon[89212]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:07:41.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:41 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:07:41.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:41 smithi170 ceph-mon[65866]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:07:41.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:41 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:07:42.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:42 smithi077 ceph-mon[84247]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:42.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:42 smithi077 ceph-mon[89212]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:42.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:42 smithi170 ceph-mon[65866]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:44.046 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:07:43 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:07:43] "GET /metrics HTTP/1.1" 200 38482 "" "Prometheus/2.43.0" 2024-05-06T14:07:44.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:44 smithi077 ceph-mon[84247]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:44.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:44 smithi077 ceph-mon[89212]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:44.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:44 smithi170 ceph-mon[65866]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:46.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:46 smithi077 ceph-mon[84247]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:46.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:46 smithi077 ceph-mon[89212]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:46.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:46 smithi170 ceph-mon[65866]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:48.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:48 smithi077 ceph-mon[84247]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:48.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:48 smithi077 ceph-mon[89212]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:48.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:48 smithi170 ceph-mon[65866]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:49.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:49 smithi077 ceph-mon[84247]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:49.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:49 smithi077 ceph-mon[89212]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:49.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:49 smithi170 ceph-mon[65866]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:50.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:50 smithi077 ceph-mon[84247]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:07:50.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:50 smithi077 ceph-mon[89212]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:07:50.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:50 smithi170 ceph-mon[65866]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:07:51.545 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:51 smithi077 ceph-mon[84247]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:51.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:51 smithi077 ceph-mon[89212]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:51.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:51 smithi170 ceph-mon[65866]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:54.046 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:07:53 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:07:53] "GET /metrics HTTP/1.1" 200 38482 "" "Prometheus/2.43.0" 2024-05-06T14:07:54.545 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:54 smithi077 ceph-mon[84247]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:54.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:54 smithi077 ceph-mon[89212]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:54.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:54 smithi170 ceph-mon[65866]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:56.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:56 smithi077 ceph-mon[84247]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:56.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:56 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:07:56.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:56 smithi077 ceph-mon[89212]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:56.547 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:56 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:07:56.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:56 smithi170 ceph-mon[65866]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:07:56.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:56 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:07:58.296 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:07:58 smithi077 ceph-mon[89212]: pgmap v358: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:58.296 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:07:58 smithi077 ceph-mon[84247]: pgmap v358: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:07:58.537 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch ps' 2024-05-06T14:07:58.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:07:58 smithi170 ceph-mon[65866]: pgmap v358: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:08:00.296 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:00 smithi077 ceph-mon[84247]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:08:00.296 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:00 smithi077 ceph-mon[89212]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:08:00.463 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-06T14:08:00.463 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.a smithi077 *:9093,9094 running (12m) 2m ago 35m 26.0M - 0.25.0 c8568f914cd2 997175e3e64e 2024-05-06T14:08:00.463 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.a smithi170 *:3000 running (12m) 2m ago 34m 81.2M - 9.4.12 2bacad6d85d8 140bb9a2760a 2024-05-06T14:08:00.463 INFO:teuthology.orchestra.run.smithi077.stdout:iscsi.foo.smithi077.muddwz smithi077 running (13m) 2m ago 34m 64.4M - 3.5 e1d6a67b021e 9353a303fb55 2024-05-06T14:08:00.463 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.x smithi170 *:8443,9283,8765 running (14m) 2m ago 39m 453M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 125521be53a3 2024-05-06T14:08:00.463 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.y smithi077 *:8443,9283,8765 running (23m) 2m ago 41m 514M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 129a14f19c52 2024-05-06T14:08:00.463 INFO:teuthology.orchestra.run.smithi077.stdout:mon.a smithi077 running (10m) 2m ago 41m 58.2M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 5e50731ccbe8 2024-05-06T14:08:00.463 INFO:teuthology.orchestra.run.smithi077.stdout:mon.b smithi170 running (11m) 2m ago 39m 49.9M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 61cd48b3d0ff 2024-05-06T14:08:00.464 INFO:teuthology.orchestra.run.smithi077.stdout:mon.c smithi077 running (10m) 2m ago 40m 47.6M 2048M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 8a473cc2ffd1 2024-05-06T14:08:00.464 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.a smithi077 *:9100 running (13m) 2m ago 35m 16.4M - 1.5.0 0da6a335fe13 67f45fa4968b 2024-05-06T14:08:00.464 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.b smithi170 *:9100 running (13m) 2m ago 35m 15.9M - 1.5.0 0da6a335fe13 02081b71a5dd 2024-05-06T14:08:00.464 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (8m) 2m ago 39m 52.6M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 fbe876a933ed 2024-05-06T14:08:00.464 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (7m) 2m ago 38m 54.0M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 6f664e567c93 2024-05-06T14:08:00.464 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (6m) 2m ago 38m 51.5M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 a4c3266b87f8 2024-05-06T14:08:00.464 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi077 running (6m) 2m ago 38m 52.5M 2992M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 7c9e6dbacb88 2024-05-06T14:08:00.464 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi170 running (5m) 2m ago 37m 51.3M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 345a6ce2c7c3 2024-05-06T14:08:00.464 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi170 running (4m) 2m ago 37m 49.1M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 fa8ea71417a6 2024-05-06T14:08:00.464 INFO:teuthology.orchestra.run.smithi077.stdout:osd.6 smithi170 running (4m) 2m ago 37m 49.6M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 1fbae25547b9 2024-05-06T14:08:00.464 INFO:teuthology.orchestra.run.smithi077.stdout:osd.7 smithi170 running (3m) 2m ago 36m 55.0M 3248M 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 82e3dd2bbdd9 2024-05-06T14:08:00.464 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.a smithi170 *:9095 running (13m) 2m ago 36m 73.0M - 2.43.0 a07b618ecd1d 10d5c513bb9b 2024-05-06T14:08:00.464 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi077.xctpgy smithi077 *:8000 running (2m) 2m ago 34m 74.2M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 bfc48a115dcd 2024-05-06T14:08:00.464 INFO:teuthology.orchestra.run.smithi077.stdout:rgw.foo.smithi170.rackmk smithi170 *:8000 running (2m) 2m ago 34m 76.1M - 19.0.0-2498-g70ce9539 e4ce8ad5d0c4 d5e5069aa8a4 2024-05-06T14:08:00.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:00 smithi170 ceph-mon[65866]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:08:01.057 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph versions' 2024-05-06T14:08:01.504 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:01 smithi077 ceph-mon[84247]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:08:01.505 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:01 smithi077 ceph-mon[89212]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:08:01.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:01 smithi170 ceph-mon[65866]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:08:02.323 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:02 smithi077 ceph-mon[84247]: from='client.64891 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:08:02.323 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:02 smithi077 ceph-mon[84247]: pgmap v360: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:08:02.323 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:02 smithi077 ceph-mon[89212]: from='client.64891 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:08:02.324 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:02 smithi077 ceph-mon[89212]: pgmap v360: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:08:02.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:02 smithi170 ceph-mon[65866]: from='client.64891 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:08:02.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:02 smithi170 ceph-mon[65866]: pgmap v360: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:08:03.074 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:08:03.075 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-05-06T14:08:03.075 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 3 2024-05-06T14:08:03.075 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:08:03.075 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-05-06T14:08:03.075 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T14:08:03.075 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:08:03.075 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-05-06T14:08:03.075 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 8 2024-05-06T14:08:03.075 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:08:03.075 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw": { 2024-05-06T14:08:03.075 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 2 2024-05-06T14:08:03.075 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-05-06T14:08:03.076 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-05-06T14:08:03.076 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)": 15 2024-05-06T14:08:03.076 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-05-06T14:08:03.076 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:08:03.372 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:03 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/4016891711' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:08:03.373 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:03 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/4016891711' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:08:03.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:03 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/4016891711' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:08:03.715 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch upgrade status' 2024-05-06T14:08:04.021 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:08:03] "GET /metrics HTTP/1.1" 200 38485 "" "Prometheus/2.43.0" 2024-05-06T14:08:04.272 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:04 smithi077 ceph-mon[84247]: pgmap v361: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:08:04.272 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:04 smithi077 ceph-mon[89212]: pgmap v361: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:08:04.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:04 smithi170 ceph-mon[65866]: pgmap v361: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:08:05.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:05 smithi077 ceph-mon[84247]: pgmap v362: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:08:05.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:05 smithi077 ceph-mon[89212]: pgmap v362: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:08:05.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:05 smithi170 ceph-mon[65866]: pgmap v362: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:08:05.593 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:08:05.593 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa", 2024-05-06T14:08:05.593 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-05-06T14:08:05.593 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading all daemon types on all hosts", 2024-05-06T14:08:05.593 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [ 2024-05-06T14:08:05.593 INFO:teuthology.orchestra.run.smithi077.stdout: "mon", 2024-05-06T14:08:05.593 INFO:teuthology.orchestra.run.smithi077.stdout: "rgw", 2024-05-06T14:08:05.593 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr", 2024-05-06T14:08:05.593 INFO:teuthology.orchestra.run.smithi077.stdout: "osd" 2024-05-06T14:08:05.593 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-05-06T14:08:05.593 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "15/21 daemons upgraded", 2024-05-06T14:08:05.593 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi077.muddwz on host smithi077 failed.", 2024-05-06T14:08:05.593 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": true 2024-05-06T14:08:05.593 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:08:06.200 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph health detail' 2024-05-06T14:08:06.295 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:06 smithi077 ceph-mon[84247]: from='client.54969 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:08:06.296 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:06 smithi077 ceph-mon[89212]: from='client.54969 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:08:06.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:06 smithi170 ceph-mon[65866]: from='client.54969 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:08:07.492 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:07 smithi077 ceph-mon[84247]: pgmap v363: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:08:07.493 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:07 smithi077 ceph-mon[89212]: pgmap v363: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:08:07.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:07 smithi170 ceph-mon[65866]: pgmap v363: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:08:08.252 INFO:teuthology.orchestra.run.smithi077.stdout:HEALTH_WARN Upgrading daemon iscsi.foo.smithi077.muddwz on host smithi077 failed. 2024-05-06T14:08:08.252 INFO:teuthology.orchestra.run.smithi077.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi077.muddwz on host smithi077 failed. 2024-05-06T14:08:08.252 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrade daemon: iscsi.foo.smithi077.muddwz: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi077.muddwz ... 2024-05-06T14:08:08.252 INFO:teuthology.orchestra.run.smithi077.stdout:Creating ceph-iscsi config... 2024-05-06T14:08:08.252 INFO:teuthology.orchestra.run.smithi077.stdout:Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/iscsi-gateway.cfg 2024-05-06T14:08:08.252 INFO:teuthology.orchestra.run.smithi077.stdout:Write file: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/iscsi.foo.smithi077.muddwz/tcmu-runner-entrypoint.sh 2024-05-06T14:08:08.252 INFO:teuthology.orchestra.run.smithi077.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d20e59cb4\x2d0bac\x2d11ef\x2dbc97\x2dc7b262605968.slice/ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service 2024-05-06T14:08:08.252 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 1 from systemctl start ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz 2024-05-06T14:08:08.253 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T14:08:08.253 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T14:08:08.253 INFO:teuthology.orchestra.run.smithi077.stdout:Traceback (most recent call last): 2024-05-06T14:08:08.253 INFO:teuthology.orchestra.run.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-05-06T14:08:08.253 INFO:teuthology.orchestra.run.smithi077.stdout: return _run_code(code, main_globals, None, 2024-05-06T14:08:08.253 INFO:teuthology.orchestra.run.smithi077.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-05-06T14:08:08.253 INFO:teuthology.orchestra.run.smithi077.stdout: exec(code, run_globals) 2024-05-06T14:08:08.253 INFO:teuthology.orchestra.run.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5618, in 2024-05-06T14:08:08.253 INFO:teuthology.orchestra.run.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 5606, in main 2024-05-06T14:08:08.253 INFO:teuthology.orchestra.run.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3096, in command_deploy_from 2024-05-06T14:08:08.253 INFO:teuthology.orchestra.run.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3131, in _common_deploy 2024-05-06T14:08:08.253 INFO:teuthology.orchestra.run.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 3151, in _deploy_daemon_container 2024-05-06T14:08:08.253 INFO:teuthology.orchestra.run.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1087, in deploy_daemon 2024-05-06T14:08:08.253 INFO:teuthology.orchestra.run.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/__main__.py", line 1223, in deploy_daemon_units 2024-05-06T14:08:08.253 INFO:teuthology.orchestra.run.smithi077.stdout: File "/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/cephadm.1beb51c47e1829c362cd9b19cf8a7b999ad8bbb9ab73d24fbfffb07828920cba/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-05-06T14:08:08.254 INFO:teuthology.orchestra.run.smithi077.stdout:RuntimeError: Failed command: systemctl start ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz: Job for ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service failed because the control process exited with error code. 2024-05-06T14:08:08.254 INFO:teuthology.orchestra.run.smithi077.stdout:See "systemctl status ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" and "journalctl -xeu ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@iscsi.foo.smithi077.muddwz.service" for details. 2024-05-06T14:08:08.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:08 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/4121201417' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T14:08:08.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:08 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/4121201417' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T14:08:08.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:08 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/4121201417' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-05-06T14:08:08.918 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-05-06T14:08:09.530 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:09 smithi077 ceph-mon[84247]: pgmap v364: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:08:09.530 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:09 smithi077 ceph-mon[89212]: pgmap v364: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:08:09.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:09 smithi170 ceph-mon[65866]: pgmap v364: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:08:10.420 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:10 smithi077 ceph-mon[84247]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:08:10.420 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:10 smithi077 ceph-mon[89212]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:08:10.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:10 smithi170 ceph-mon[65866]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:08:10.921 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-05-06T14:08:11.514 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:11 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/1550275422' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:08:11.514 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:11 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:08:11.514 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:11 smithi077 ceph-mon[84247]: pgmap v365: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:08:11.515 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:11 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/1550275422' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:08:11.515 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:11 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:08:11.515 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:11 smithi077 ceph-mon[89212]: pgmap v365: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:08:11.532 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-05-06T14:08:11.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:11 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/1550275422' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:08:11.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:11 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:08:11.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:11 smithi170 ceph-mon[65866]: pgmap v365: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:08:13.549 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev)" 2024-05-06T14:08:13.891 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:13 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:08:13] "GET /metrics HTTP/1.1" 200 38485 "" "Prometheus/2.43.0" 2024-05-06T14:08:14.225 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -e sha1=70ce9539b62a15b9de429c1f0e91a1e012b099aa -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-05-06T14:08:14.529 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:14 smithi077 ceph-mon[84247]: pgmap v366: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:08:14.529 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:14 smithi077 ceph-mon[84247]: from='client.? 172.21.15.77:0/781275477' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:08:14.529 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:14 smithi077 ceph-mon[89212]: pgmap v366: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:08:14.529 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:14 smithi077 ceph-mon[89212]: from='client.? 172.21.15.77:0/781275477' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:08:14.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:14 smithi170 ceph-mon[65866]: pgmap v366: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:08:14.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:14 smithi170 ceph-mon[65866]: from='client.? 172.21.15.77:0/781275477' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-05-06T14:08:16.167 INFO:teuthology.orchestra.run.smithi077.stdout:osd 8 2m ago - 2024-05-06T14:08:16.545 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:16 smithi077 ceph-mon[84247]: pgmap v367: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:08:16.545 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:16 smithi077 ceph-mon[89212]: pgmap v367: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:08:16.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:16 smithi170 ceph-mon[65866]: pgmap v367: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:08:16.865 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-06T14:08:16.876 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi077.front.sepia.ceph.com 2024-05-06T14:08:16.877 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- bash -c 'ceph orch upgrade ls' 2024-05-06T14:08:17.522 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:17 smithi077 ceph-mon[84247]: from='client.64927 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:08:17.522 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:17 smithi077 ceph-mon[89212]: from='client.64927 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:08:17.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:17 smithi170 ceph-mon[65866]: from='client.64927 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:08:18.389 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:18 smithi077 ceph-mon[84247]: pgmap v368: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:08:18.390 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:18 smithi077 ceph-mon[89212]: pgmap v368: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:08:18.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:18 smithi170 ceph-mon[65866]: pgmap v368: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:08:19.906 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-05-06T14:08:19.906 INFO:teuthology.orchestra.run.smithi077.stdout: "image": "quay.io/ceph/ceph", 2024-05-06T14:08:19.906 INFO:teuthology.orchestra.run.smithi077.stdout: "registry": "quay.io", 2024-05-06T14:08:19.907 INFO:teuthology.orchestra.run.smithi077.stdout: "bare_image": "ceph/ceph", 2024-05-06T14:08:19.907 INFO:teuthology.orchestra.run.smithi077.stdout: "versions": [] 2024-05-06T14:08:19.907 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-05-06T14:08:20.545 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:20 smithi077 ceph-mon[84247]: from='client.64933 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:08:20.545 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:20 smithi077 ceph-mon[84247]: pgmap v369: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:08:20.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:20 smithi077 ceph-mon[89212]: from='client.64933 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:08:20.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:20 smithi077 ceph-mon[89212]: pgmap v369: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:08:20.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:20 smithi170 ceph-mon[65866]: from='client.64933 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:08:20.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:20 smithi170 ceph-mon[65866]: pgmap v369: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:08:20.597 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-05-06T14:08:21.545 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:21 smithi077 ceph-mon[84247]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:08:21.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:21 smithi077 ceph-mon[89212]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:08:21.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:21 smithi170 ceph-mon[65866]: from='client.64816 -' entity='client.iscsi.foo.smithi077.muddwz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-06T14:08:22.545 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:22 smithi077 ceph-mon[84247]: pgmap v370: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:08:22.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:22 smithi077 ceph-mon[89212]: pgmap v370: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:08:22.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:22 smithi170 ceph-mon[65866]: pgmap v370: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:08:23.419 INFO:teuthology.orchestra.run.smithi077.stdout: "16.2.0", 2024-05-06T14:08:23.545 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:23 smithi077 ceph-mon[84247]: from='client.64939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:08:23.545 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:23 smithi077 ceph-mon[84247]: pgmap v371: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:08:23.545 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:23 smithi077 ceph-mon[89212]: from='client.64939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:08:23.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:23 smithi077 ceph-mon[89212]: pgmap v371: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:08:23.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:23 smithi170 ceph-mon[65866]: from='client.64939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:08:23.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:23 smithi170 ceph-mon[65866]: pgmap v371: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-06T14:08:24.000 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:23 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: ::ffff:172.21.15.170 - - [06/May/2024:14:08:23] "GET /metrics HTTP/1.1" 200 38485 "" "Prometheus/2.43.0" 2024-05-06T14:08:24.025 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-05-06T14:08:26.545 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:26 smithi077 ceph-mon[84247]: pgmap v372: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:08:26.546 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:26 smithi077 ceph-mon[84247]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:08:26.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:26 smithi077 ceph-mon[89212]: pgmap v372: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:08:26.546 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:26 smithi077 ceph-mon[89212]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:08:26.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:26 smithi170 ceph-mon[65866]: pgmap v372: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-06T14:08:26.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:26 smithi170 ceph-mon[65866]: from='mgr.25849 172.21.15.77:0/4293661156' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-06T14:08:26.935 INFO:teuthology.orchestra.run.smithi077.stdout: "v16.2.2", 2024-05-06T14:08:26.935 INFO:teuthology.orchestra.run.smithi077.stdout: "v16.2.2-20210505", 2024-05-06T14:08:27.533 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:27 smithi077 ceph-mon[84247]: from='client.64945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:08:27.533 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:27 smithi077 ceph-mon[89212]: from='client.64945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:08:27.557 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-05-06T14:08:27.566 INFO:tasks.cephadm:Teardown begin 2024-05-06T14:08:27.567 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-06T14:08:27.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:27 smithi170 ceph-mon[65866]: from='client.64945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-06T14:08:27.594 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-06T14:08:27.625 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-05-06T14:08:27.625 DEBUG:teuthology.orchestra.run.smithi077:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-06T14:08:27.650 DEBUG:teuthology.orchestra.run.smithi170:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-06T14:08:27.682 INFO:tasks.cephadm:Stopping all daemons... 2024-05-06T14:08:27.682 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-05-06T14:08:27.682 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mon.a 2024-05-06T14:08:27.795 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:27 smithi077 systemd[1]: Stopping Ceph mon.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:08:28.295 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:27 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-a[84243]: 2024-05-06T14:08:27.976+0000 7f321635a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:08:28.296 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:27 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-a[84243]: 2024-05-06T14:08:27.976+0000 7f321635a700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-05-06T14:08:28.296 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:28 smithi077 podman[131041]: 2024-05-06 14:08:28.140240402 +0000 UTC m=+0.261470551 container died 5e50731ccbe87044c3f1a3e8f154130cf0f595ef2d3249272436cba1a9a5bf51 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-a, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-05-06T14:08:28.296 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:27 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:14:08:27] ENGINE Bus STOPPING 2024-05-06T14:08:28.296 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:27 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:14:08:27] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-06T14:08:28.296 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:27 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:14:08:27] ENGINE Bus STOPPED 2024-05-06T14:08:28.296 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:27 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:14:08:27] ENGINE Bus STARTING 2024-05-06T14:08:28.296 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:28 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: CherryPy Checker: 2024-05-06T14:08:28.297 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:28 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: The Application mounted at '' has an empty config. 2024-05-06T14:08:28.297 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:28 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T14:08:28.297 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:28 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:14:08:28] ENGINE Serving on http://:::9283 2024-05-06T14:08:28.297 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:28 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:14:08:28] ENGINE Bus STARTED 2024-05-06T14:08:28.795 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:28 smithi077 podman[131041]: 2024-05-06 14:08:28.585291847 +0000 UTC m=+0.706521992 container cleanup 5e50731ccbe87044c3f1a3e8f154130cf0f595ef2d3249272436cba1a9a5bf51 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-a, org.label-schema.schema-version=1.0, RELEASE=squid-70ce953, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T14:08:28.795 INFO:journalctl@ceph.mon.a.smithi077.stdout:May 06 14:08:28 smithi077 bash[131041]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-a 2024-05-06T14:08:29.154 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mon.a.service' 2024-05-06T14:08:29.202 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T14:08:29.202 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-05-06T14:08:29.202 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-05-06T14:08:29.202 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mon.c 2024-05-06T14:08:29.295 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:29 smithi077 systemd[1]: Stopping Ceph mon.c for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:08:29.796 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:29 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-c[89207]: 2024-05-06T14:08:29.481+0000 7fc492437700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:08:29.796 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:29 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-c[89207]: 2024-05-06T14:08:29.481+0000 7fc492437700 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-05-06T14:08:29.796 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:29 smithi077 podman[131174]: 2024-05-06 14:08:29.676347657 +0000 UTC m=+0.292237691 container died 8a473cc2ffd10a1f9daa349f5b49d6e9dc2fac143a6c9c959e3e8505e0d0fddd (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-05-06T14:08:29.796 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:29 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:14:08:29] ENGINE Bus STOPPING 2024-05-06T14:08:29.796 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:29 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:14:08:29] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-06T14:08:29.796 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:29 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:14:08:29] ENGINE Bus STOPPED 2024-05-06T14:08:29.796 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:29 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:14:08:29] ENGINE Bus STARTING 2024-05-06T14:08:29.797 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:29 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:14:08:29] ENGINE Serving on http://:::9283 2024-05-06T14:08:29.797 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:29 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: [06/May/2024:14:08:29] ENGINE Bus STARTED 2024-05-06T14:08:30.193 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:30 smithi077 podman[131174]: 2024-05-06 14:08:30.015626224 +0000 UTC m=+0.631516254 container cleanup 8a473cc2ffd10a1f9daa349f5b49d6e9dc2fac143a6c9c959e3e8505e0d0fddd (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-05-06T14:08:30.193 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:30 smithi077 bash[131174]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-c 2024-05-06T14:08:30.430 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mon.c.service' 2024-05-06T14:08:30.450 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:30 smithi077 podman[131199]: 2024-05-06 14:08:30.200331343 +0000 UTC m=+0.520910834 container remove 8a473cc2ffd10a1f9daa349f5b49d6e9dc2fac143a6c9c959e3e8505e0d0fddd (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-c, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430) 2024-05-06T14:08:30.450 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:30 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mon.c.service: Deactivated successfully. 2024-05-06T14:08:30.450 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:30 smithi077 systemd[1]: Stopped Ceph mon.c for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T14:08:30.450 INFO:journalctl@ceph.mon.c.smithi077.stdout:May 06 14:08:30 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mon.c.service: Consumed 6.272s CPU time. 2024-05-06T14:08:30.463 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T14:08:30.463 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-05-06T14:08:30.463 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-05-06T14:08:30.464 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mon.b 2024-05-06T14:08:30.796 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:30 smithi170 systemd[1]: Stopping Ceph mon.b for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:08:31.069 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:30 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-b[65860]: 2024-05-06T14:08:30.801+0000 7ff963a7c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:08:31.070 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:30 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-b[65860]: 2024-05-06T14:08:30.801+0000 7ff963a7c700 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-05-06T14:08:31.070 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:30 smithi170 podman[99363]: 2024-05-06 14:08:30.874115377 +0000 UTC m=+0.185715343 container died 61cd48b3d0ff22847979c7591e6adaec1ad30037835ff47d257f3146ffc3d053 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-b, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-70ce953) 2024-05-06T14:08:31.569 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:31 smithi170 podman[99363]: 2024-05-06 14:08:31.327839117 +0000 UTC m=+0.639439079 container cleanup 61cd48b3d0ff22847979c7591e6adaec1ad30037835ff47d257f3146ffc3d053 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-b, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-70ce953, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T14:08:31.570 INFO:journalctl@ceph.mon.b.smithi170.stdout:May 06 14:08:31 smithi170 bash[99363]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mon-b 2024-05-06T14:08:31.746 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mon.b.service' 2024-05-06T14:08:31.789 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T14:08:31.789 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-05-06T14:08:31.790 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-05-06T14:08:31.790 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mgr.y 2024-05-06T14:08:32.112 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:31 smithi077 systemd[1]: Stopping Ceph mgr.y for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:08:32.481 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:32 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T14:08:32.118+0000 7f5d08d43700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:08:32.481 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:32 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y[60776]: 2024-05-06T14:08:32.118+0000 7f5d08d43700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-05-06T14:08:32.481 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:32 smithi077 podman[131306]: 2024-05-06 14:08:32.15055383 +0000 UTC m=+0.162276549 container stop 129a14f19c525bdbf648f43c0353e1dd88fe1c91e2d821bf08c1e2b5ea186f7b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240430, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-05-06T14:08:32.481 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:32 smithi077 podman[131306]: 2024-05-06 14:08:32.176078239 +0000 UTC m=+0.187800959 container died 129a14f19c525bdbf648f43c0353e1dd88fe1c91e2d821bf08c1e2b5ea186f7b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T14:08:32.790 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:32 smithi077 podman[131306]: 2024-05-06 14:08:32.620951518 +0000 UTC m=+0.632674239 container cleanup 129a14f19c525bdbf648f43c0353e1dd88fe1c91e2d821bf08c1e2b5ea186f7b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y, org.label-schema.license=GPLv2, RELEASE=squid-70ce953, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240430) 2024-05-06T14:08:32.790 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:32 smithi077 bash[131306]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y 2024-05-06T14:08:33.018 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mgr.y.service' 2024-05-06T14:08:33.042 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:32 smithi077 podman[131321]: 2024-05-06 14:08:32.797531866 +0000 UTC m=+0.637105277 container remove 129a14f19c525bdbf648f43c0353e1dd88fe1c91e2d821bf08c1e2b5ea186f7b (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240430) 2024-05-06T14:08:33.042 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:33 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mgr.y.service: Deactivated successfully. 2024-05-06T14:08:33.042 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:33 smithi077 systemd[1]: Stopped Ceph mgr.y for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T14:08:33.042 INFO:journalctl@ceph.mgr.y.smithi077.stdout:May 06 14:08:33 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mgr.y.service: Consumed 1min 14.811s CPU time. 2024-05-06T14:08:33.055 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T14:08:33.055 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-05-06T14:08:33.055 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-05-06T14:08:33.055 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mgr.x 2024-05-06T14:08:33.398 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 14:08:33 smithi170 systemd[1]: Stopping Ceph mgr.x for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:08:33.685 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 14:08:33 smithi170 podman[99487]: 2024-05-06 14:08:33.405309756 +0000 UTC m=+0.154490157 container died 125521be53a3c22765c4036f2d53d0ccb5fb2b05da7976421dbaf0b94c474aec (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-70ce953, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0) 2024-05-06T14:08:34.069 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 14:08:33 smithi170 podman[99487]: 2024-05-06 14:08:33.883453974 +0000 UTC m=+0.632634367 container cleanup 125521be53a3c22765c4036f2d53d0ccb5fb2b05da7976421dbaf0b94c474aec (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, RELEASE=squid-70ce953) 2024-05-06T14:08:34.069 INFO:journalctl@ceph.mgr.x.smithi170.stdout:May 06 14:08:33 smithi170 bash[99487]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-mgr-x 2024-05-06T14:08:34.453 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@mgr.x.service' 2024-05-06T14:08:34.485 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T14:08:34.486 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-05-06T14:08:34.486 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-05-06T14:08:34.486 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.0 2024-05-06T14:08:34.795 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 14:08:34 smithi077 systemd[1]: Stopping Ceph osd.0 for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:08:35.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 14:08:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0[96694]: 2024-05-06T14:08:34.808+0000 7f7cbe078700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:08:35.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 14:08:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0[96694]: 2024-05-06T14:08:34.808+0000 7f7cbe078700 -1 osd.0 160 *** Got signal Terminated *** 2024-05-06T14:08:35.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 14:08:34 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0[96694]: 2024-05-06T14:08:34.808+0000 7f7cbe078700 -1 osd.0 160 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T14:08:40.487 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 14:08:40 smithi077 podman[131438]: 2024-05-06 14:08:40.225171941 +0000 UTC m=+5.542892832 container died fbe876a933edd8dfb84a48e5b5e09a4a937b71085a458772846a4171c9bdf5d8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-70ce953, org.label-schema.build-date=20240430) 2024-05-06T14:08:40.795 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 14:08:40 smithi077 podman[131438]: 2024-05-06 14:08:40.584399705 +0000 UTC m=+5.902120593 container cleanup fbe876a933edd8dfb84a48e5b5e09a4a937b71085a458772846a4171c9bdf5d8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-05-06T14:08:40.795 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 14:08:40 smithi077 bash[131438]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0 2024-05-06T14:08:41.198 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 14:08:40 smithi077 podman[131455]: 2024-05-06 14:08:40.852741537 +0000 UTC m=+0.620075827 container remove fbe876a933edd8dfb84a48e5b5e09a4a937b71085a458772846a4171c9bdf5d8 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-70ce953, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T14:08:41.457 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 14:08:41 smithi077 podman[131546]: 2024-05-06 14:08:41.109111617 +0000 UTC m=+0.018206105 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:08:41.458 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 14:08:41 smithi077 podman[131546]: 2024-05-06 14:08:41.253987777 +0000 UTC m=+0.163082252 container create 469d54786a3db5ce9a65f364c0ea2d43fb2c9dc02c89820102b9795b3c94f76d (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, ceph=True, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-05-06T14:08:41.458 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 14:08:41 smithi077 podman[131546]: 2024-05-06 14:08:41.464869631 +0000 UTC m=+0.373964107 container init 469d54786a3db5ce9a65f364c0ea2d43fb2c9dc02c89820102b9795b3c94f76d (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-06T14:08:41.752 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 14:08:41 smithi077 podman[131546]: 2024-05-06 14:08:41.468253255 +0000 UTC m=+0.377347739 container start 469d54786a3db5ce9a65f364c0ea2d43fb2c9dc02c89820102b9795b3c94f76d (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T14:08:41.752 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 14:08:41 smithi077 podman[131546]: 2024-05-06 14:08:41.523375181 +0000 UTC m=+0.432469670 container attach 469d54786a3db5ce9a65f364c0ea2d43fb2c9dc02c89820102b9795b3c94f76d (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-05-06T14:08:42.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:May 06 14:08:41 smithi077 podman[131546]: 2024-05-06 14:08:41.759726864 +0000 UTC m=+0.668821350 container died 469d54786a3db5ce9a65f364c0ea2d43fb2c9dc02c89820102b9795b3c94f76d (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T14:08:42.491 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.0.service' 2024-05-06T14:08:42.639 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T14:08:42.639 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-05-06T14:08:42.639 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-05-06T14:08:42.639 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.1 2024-05-06T14:08:42.915 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:08:42 smithi077 systemd[1]: Stopping Ceph osd.1 for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:08:43.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:08:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1[101871]: 2024-05-06T14:08:42.921+0000 7fb6c63c0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:08:43.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:08:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1[101871]: 2024-05-06T14:08:42.921+0000 7fb6c63c0700 -1 osd.1 160 *** Got signal Terminated *** 2024-05-06T14:08:43.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:08:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1[101871]: 2024-05-06T14:08:42.921+0000 7fb6c63c0700 -1 osd.1 160 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T14:08:48.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:08:48 smithi077 podman[131652]: 2024-05-06 14:08:48.22851849 +0000 UTC m=+5.406570150 container died 6f664e567c93b2d2ed95c3440466046cfeeaacc7282b685296ace3e8528a17a2 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-70ce953, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-05-06T14:08:48.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:08:48 smithi077 podman[131652]: 2024-05-06 14:08:48.605667894 +0000 UTC m=+5.783719553 container cleanup 6f664e567c93b2d2ed95c3440466046cfeeaacc7282b685296ace3e8528a17a2 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T14:08:48.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:08:48 smithi077 bash[131652]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1 2024-05-06T14:08:49.212 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:08:48 smithi077 podman[131670]: 2024-05-06 14:08:48.907118593 +0000 UTC m=+0.670471979 container remove 6f664e567c93b2d2ed95c3440466046cfeeaacc7282b685296ace3e8528a17a2 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1, org.label-schema.build-date=20240430, RELEASE=squid-70ce953, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True) 2024-05-06T14:08:49.496 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:08:49 smithi077 podman[131762]: 2024-05-06 14:08:49.122328341 +0000 UTC m=+0.018147727 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:08:49.496 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:08:49 smithi077 podman[131762]: 2024-05-06 14:08:49.253030349 +0000 UTC m=+0.148849779 container create 33bd0d24cfe5772768890f1ae962738c47ead2d97bd3566f8b0b1eb9bcc60d23 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0) 2024-05-06T14:08:49.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:08:49 smithi077 podman[131762]: 2024-05-06 14:08:49.503847374 +0000 UTC m=+0.399666759 container init 33bd0d24cfe5772768890f1ae962738c47ead2d97bd3566f8b0b1eb9bcc60d23 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-05-06T14:08:49.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:08:49 smithi077 podman[131762]: 2024-05-06 14:08:49.507062376 +0000 UTC m=+0.402881763 container start 33bd0d24cfe5772768890f1ae962738c47ead2d97bd3566f8b0b1eb9bcc60d23 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-06T14:08:49.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:08:49 smithi077 podman[131762]: 2024-05-06 14:08:49.586423938 +0000 UTC m=+0.482243352 container attach 33bd0d24cfe5772768890f1ae962738c47ead2d97bd3566f8b0b1eb9bcc60d23 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-70ce953, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T14:08:50.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:08:49 smithi077 podman[131762]: 2024-05-06 14:08:49.799040514 +0000 UTC m=+0.694859890 container died 33bd0d24cfe5772768890f1ae962738c47ead2d97bd3566f8b0b1eb9bcc60d23 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1-deactivate, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True) 2024-05-06T14:08:50.326 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.1.service' 2024-05-06T14:08:50.347 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:08:50 smithi077 podman[131788]: 2024-05-06 14:08:50.307158027 +0000 UTC m=+0.502802983 container remove 33bd0d24cfe5772768890f1ae962738c47ead2d97bd3566f8b0b1eb9bcc60d23 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-70ce953, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-06T14:08:50.348 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:08:50 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.1.service: Deactivated successfully. 2024-05-06T14:08:50.348 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:08:50 smithi077 systemd[1]: Stopped Ceph osd.1 for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T14:08:50.348 INFO:journalctl@ceph.osd.1.smithi077.stdout:May 06 14:08:50 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.1.service: Consumed 6.726s CPU time. 2024-05-06T14:08:50.363 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T14:08:50.363 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-05-06T14:08:50.363 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-05-06T14:08:50.363 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.2 2024-05-06T14:08:50.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:08:50 smithi077 systemd[1]: Stopping Ceph osd.2 for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:08:51.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:08:50 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2[108048]: 2024-05-06T14:08:50.693+0000 7fb20095e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:08:51.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:08:50 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2[108048]: 2024-05-06T14:08:50.693+0000 7fb20095e700 -1 osd.2 160 *** Got signal Terminated *** 2024-05-06T14:08:51.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:08:50 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2[108048]: 2024-05-06T14:08:50.693+0000 7fb20095e700 -1 osd.2 160 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T14:08:51.069 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 14:08:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T14:08:50.692Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.77:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.77:8765: connect: connection refused" 2024-05-06T14:08:51.069 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 14:08:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T14:08:50.692Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.77:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.77:8765: connect: connection refused" 2024-05-06T14:08:51.069 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 14:08:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T14:08:50.694Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.77:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.77:8765: connect: connection refused" 2024-05-06T14:08:51.069 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 14:08:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T14:08:50.694Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.77:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.77:8765: connect: connection refused" 2024-05-06T14:08:51.070 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 14:08:50 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T14:08:50.694Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.77:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.77:8765: connect: connection refused" 2024-05-06T14:08:56.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:08:55 smithi077 podman[131867]: 2024-05-06 14:08:55.998101169 +0000 UTC m=+5.416722552 container died a4c3266b87f88c36d6b9f7be446856c82df87e390d25e49f7b58f32b163d12d3 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T14:08:56.637 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:08:56 smithi077 podman[131867]: 2024-05-06 14:08:56.418683535 +0000 UTC m=+5.837304915 container cleanup a4c3266b87f88c36d6b9f7be446856c82df87e390d25e49f7b58f32b163d12d3 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-05-06T14:08:56.637 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:08:56 smithi077 bash[131867]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2 2024-05-06T14:08:57.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:08:56 smithi077 podman[131881]: 2024-05-06 14:08:56.645010103 +0000 UTC m=+0.640028862 container remove a4c3266b87f88c36d6b9f7be446856c82df87e390d25e49f7b58f32b163d12d3 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-05-06T14:08:57.407 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:08:57 smithi077 podman[131977]: 2024-05-06 14:08:56.990207942 +0000 UTC m=+0.018052460 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:08:57.408 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:08:57 smithi077 podman[131977]: 2024-05-06 14:08:57.104198704 +0000 UTC m=+0.132043193 container create c1d20a4f16636c98ef12db2b57c8972363421ef764630928ef5c65de77b23723 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-70ce953, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-05-06T14:08:57.408 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:08:57 smithi077 podman[131977]: 2024-05-06 14:08:57.315549777 +0000 UTC m=+0.343394279 container init c1d20a4f16636c98ef12db2b57c8972363421ef764630928ef5c65de77b23723 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, ceph=True, org.label-schema.license=GPLv2) 2024-05-06T14:08:57.408 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:08:57 smithi077 podman[131977]: 2024-05-06 14:08:57.318948105 +0000 UTC m=+0.346792600 container start c1d20a4f16636c98ef12db2b57c8972363421ef764630928ef5c65de77b23723 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-05-06T14:08:57.408 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:08:57 smithi077 podman[131977]: 2024-05-06 14:08:57.415162289 +0000 UTC m=+0.443006808 container attach c1d20a4f16636c98ef12db2b57c8972363421ef764630928ef5c65de77b23723 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-05-06T14:08:57.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:May 06 14:08:57 smithi077 podman[131977]: 2024-05-06 14:08:57.619079425 +0000 UTC m=+0.646923929 container died c1d20a4f16636c98ef12db2b57c8972363421ef764630928ef5c65de77b23723 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-2-deactivate, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T14:08:58.193 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.2.service' 2024-05-06T14:08:58.232 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T14:08:58.232 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-05-06T14:08:58.233 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-05-06T14:08:58.233 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.3 2024-05-06T14:08:58.545 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:08:58 smithi077 systemd[1]: Stopping Ceph osd.3 for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:08:59.045 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:08:58 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3[114507]: 2024-05-06T14:08:58.582+0000 7f8ab512d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:08:59.045 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:08:58 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3[114507]: 2024-05-06T14:08:58.582+0000 7f8ab512d700 -1 osd.3 160 *** Got signal Terminated *** 2024-05-06T14:08:59.045 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:08:58 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3[114507]: 2024-05-06T14:08:58.583+0000 7f8ab512d700 -1 osd.3 160 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T14:09:01.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:00 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:00.612+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:01.545 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:09:01 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3[114507]: 2024-05-06T14:09:01.297+0000 7f8ab1721700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:35.669420+0000 front 2024-05-06T14:08:35.669437+0000 (oldest deadline 2024-05-06T14:09:00.969429+0000) 2024-05-06T14:09:01.569 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4[76044]: 2024-05-06T14:09:01.225+0000 7fc84d18a700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.987048+0000 front 2024-05-06T14:08:36.987268+0000 (oldest deadline 2024-05-06T14:09:01.087113+0000) 2024-05-06T14:09:02.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:01 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:01.583+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:02.545 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:09:02 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3[114507]: 2024-05-06T14:09:02.271+0000 7f8ab1721700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:35.669420+0000 front 2024-05-06T14:08:35.669437+0000 (oldest deadline 2024-05-06T14:09:00.969429+0000) 2024-05-06T14:09:02.552 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:02 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4[76044]: 2024-05-06T14:09:02.229+0000 7fc84d18a700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.987048+0000 front 2024-05-06T14:08:36.987268+0000 (oldest deadline 2024-05-06T14:09:01.087113+0000) 2024-05-06T14:09:02.819 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:02 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:02.559+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:03.525 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:03 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4[76044]: 2024-05-06T14:09:03.232+0000 7fc84d18a700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.987048+0000 front 2024-05-06T14:08:36.987268+0000 (oldest deadline 2024-05-06T14:09:01.087113+0000) 2024-05-06T14:09:03.549 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:09:03 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3[114507]: 2024-05-06T14:09:03.230+0000 7f8ab1721700 -1 osd.3 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:35.669420+0000 front 2024-05-06T14:08:35.669437+0000 (oldest deadline 2024-05-06T14:09:00.969429+0000) 2024-05-06T14:09:03.819 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:03 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:03.531+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:04.045 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:09:03 smithi077 podman[132083]: 2024-05-06 14:09:03.914007591 +0000 UTC m=+5.444740038 container died 7c9e6dbacb8830e20649677a448f1e6709c4d3fffe0d16cd8d79a0508e0a75a5 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T14:09:04.423 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:09:04 smithi077 podman[132083]: 2024-05-06 14:09:04.246473118 +0000 UTC m=+5.777205550 container cleanup 7c9e6dbacb8830e20649677a448f1e6709c4d3fffe0d16cd8d79a0508e0a75a5 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430) 2024-05-06T14:09:04.423 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:09:04 smithi077 bash[132083]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3 2024-05-06T14:09:04.558 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:04 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4[76044]: 2024-05-06T14:09:04.240+0000 7fc84d18a700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.987048+0000 front 2024-05-06T14:08:36.987268+0000 (oldest deadline 2024-05-06T14:09:01.087113+0000) 2024-05-06T14:09:04.676 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:09:04 smithi077 podman[132109]: 2024-05-06 14:09:04.431094017 +0000 UTC m=+0.521223595 container remove 7c9e6dbacb8830e20649677a448f1e6709c4d3fffe0d16cd8d79a0508e0a75a5 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-06T14:09:04.819 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:04 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:09:04.660+0000 7f34b81ab700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.210053+0000 front 2024-05-06T14:08:39.209958+0000 (oldest deadline 2024-05-06T14:09:04.509987+0000) 2024-05-06T14:09:04.819 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:04 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:04.565+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:05.004 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:09:04 smithi077 podman[132199]: 2024-05-06 14:09:04.783745726 +0000 UTC m=+0.018555792 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:09:05.004 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:09:04 smithi077 podman[132199]: 2024-05-06 14:09:04.89882892 +0000 UTC m=+0.133638962 container create b34b6f7b75344b0619d1867586f40921f71a322c7abed849415f05cf242c92d2 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3-deactivate, RELEASE=squid-70ce953, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True) 2024-05-06T14:09:05.295 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:09:05 smithi077 podman[132199]: 2024-05-06 14:09:05.126723088 +0000 UTC m=+0.361533139 container init b34b6f7b75344b0619d1867586f40921f71a322c7abed849415f05cf242c92d2 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-05-06T14:09:05.295 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:09:05 smithi077 podman[132199]: 2024-05-06 14:09:05.12998873 +0000 UTC m=+0.364798773 container start b34b6f7b75344b0619d1867586f40921f71a322c7abed849415f05cf242c92d2 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=squid-70ce953, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T14:09:05.295 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:09:05 smithi077 podman[132199]: 2024-05-06 14:09:05.184821697 +0000 UTC m=+0.419631740 container attach b34b6f7b75344b0619d1867586f40921f71a322c7abed849415f05cf242c92d2 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-05-06T14:09:05.552 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:05 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4[76044]: 2024-05-06T14:09:05.260+0000 7fc84d18a700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.987048+0000 front 2024-05-06T14:08:36.987268+0000 (oldest deadline 2024-05-06T14:09:01.087113+0000) 2024-05-06T14:09:05.723 INFO:journalctl@ceph.osd.3.smithi077.stdout:May 06 14:09:05 smithi077 podman[132199]: 2024-05-06 14:09:05.419543785 +0000 UTC m=+0.654353830 container died b34b6f7b75344b0619d1867586f40921f71a322c7abed849415f05cf242c92d2 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953) 2024-05-06T14:09:05.819 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:05 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:09:05.680+0000 7f34b81ab700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.210053+0000 front 2024-05-06T14:08:39.209958+0000 (oldest deadline 2024-05-06T14:09:04.509987+0000) 2024-05-06T14:09:05.819 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:05 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:05.559+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:06.009 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.3.service' 2024-05-06T14:09:06.046 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T14:09:06.046 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-05-06T14:09:06.046 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-05-06T14:09:06.046 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.4 2024-05-06T14:09:06.362 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:06.275+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:06.362 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:06 smithi170 systemd[1]: Stopping Ceph osd.4 for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:09:06.362 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4[76044]: 2024-05-06T14:09:06.297+0000 7fc84d18a700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.987048+0000 front 2024-05-06T14:08:36.987268+0000 (oldest deadline 2024-05-06T14:09:01.087113+0000) 2024-05-06T14:09:06.362 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4[76044]: 2024-05-06T14:09:06.368+0000 7fc850b96700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:09:06.362 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4[76044]: 2024-05-06T14:09:06.368+0000 7fc850b96700 -1 osd.4 160 *** Got signal Terminated *** 2024-05-06T14:09:06.362 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4[76044]: 2024-05-06T14:09:06.368+0000 7fc850b96700 -1 osd.4 160 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T14:09:06.715 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:06.537+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:07.069 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:06 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:09:06.722+0000 7f34b81ab700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.210053+0000 front 2024-05-06T14:08:39.209958+0000 (oldest deadline 2024-05-06T14:09:04.509987+0000) 2024-05-06T14:09:07.569 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:07 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:07.531+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:07.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:07 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:07.308+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:07.569 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:07 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4[76044]: 2024-05-06T14:09:07.324+0000 7fc84d18a700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.987048+0000 front 2024-05-06T14:08:36.987268+0000 (oldest deadline 2024-05-06T14:09:01.087113+0000) 2024-05-06T14:09:08.069 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:07 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:09:07.719+0000 7f34b81ab700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.210053+0000 front 2024-05-06T14:08:39.209958+0000 (oldest deadline 2024-05-06T14:09:04.509987+0000) 2024-05-06T14:09:08.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:08.319+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:08.569 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4[76044]: 2024-05-06T14:09:08.315+0000 7fc84d18a700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.987048+0000 front 2024-05-06T14:08:36.987268+0000 (oldest deadline 2024-05-06T14:09:01.087113+0000) 2024-05-06T14:09:08.569 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:08.557+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:09.069 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:09:08.714+0000 7f34b81ab700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.210053+0000 front 2024-05-06T14:08:39.209958+0000 (oldest deadline 2024-05-06T14:09:04.509987+0000) 2024-05-06T14:09:09.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:09.270+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:09.569 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4[76044]: 2024-05-06T14:09:09.338+0000 7fc84d18a700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.987048+0000 front 2024-05-06T14:08:36.987268+0000 (oldest deadline 2024-05-06T14:09:01.087113+0000) 2024-05-06T14:09:09.569 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4[76044]: 2024-05-06T14:09:09.338+0000 7fc84d18a700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:46.288936+0000 front 2024-05-06T14:08:46.288913+0000 (oldest deadline 2024-05-06T14:09:09.188753+0000) 2024-05-06T14:09:10.069 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:09:09.759+0000 7f34b81ab700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.210053+0000 front 2024-05-06T14:08:39.209958+0000 (oldest deadline 2024-05-06T14:09:04.509987+0000) 2024-05-06T14:09:10.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:09 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:09.607+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:10.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:10 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:10.248+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:10.569 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:10 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4[76044]: 2024-05-06T14:09:10.294+0000 7fc84d18a700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.987048+0000 front 2024-05-06T14:08:36.987268+0000 (oldest deadline 2024-05-06T14:09:01.087113+0000) 2024-05-06T14:09:10.569 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:10 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4[76044]: 2024-05-06T14:09:10.294+0000 7fc84d18a700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:46.288936+0000 front 2024-05-06T14:08:46.288913+0000 (oldest deadline 2024-05-06T14:09:09.188753+0000) 2024-05-06T14:09:11.069 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:10 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:09:10.795+0000 7f34b81ab700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.210053+0000 front 2024-05-06T14:08:39.209958+0000 (oldest deadline 2024-05-06T14:09:04.509987+0000) 2024-05-06T14:09:11.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:10 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:10.642+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:11.569 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:11 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4[76044]: 2024-05-06T14:09:11.250+0000 7fc84d18a700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.987048+0000 front 2024-05-06T14:08:36.987268+0000 (oldest deadline 2024-05-06T14:09:01.087113+0000) 2024-05-06T14:09:11.569 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:11 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4[76044]: 2024-05-06T14:09:11.250+0000 7fc84d18a700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:46.288936+0000 front 2024-05-06T14:08:46.288913+0000 (oldest deadline 2024-05-06T14:09:09.188753+0000) 2024-05-06T14:09:11.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:11 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:11.289+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:11.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:11 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:11.289+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:45.633819+0000 front 2024-05-06T14:08:45.633749+0000 (oldest deadline 2024-05-06T14:09:10.933542+0000) 2024-05-06T14:09:12.029 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:11 smithi170 podman[99624]: 2024-05-06 14:09:11.74615816 +0000 UTC m=+5.487210341 container died 345a6ce2c7c339d4812ff990f8f94c3ec5b5ad8a68105243c1aa74c4cf449093 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240430) 2024-05-06T14:09:12.030 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:11 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:09:11.799+0000 7f34b81ab700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.210053+0000 front 2024-05-06T14:08:39.209958+0000 (oldest deadline 2024-05-06T14:09:04.509987+0000) 2024-05-06T14:09:12.030 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:11 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:11.594+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:12.030 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:11 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:11.594+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:12.295 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:12 smithi170 podman[99624]: 2024-05-06 14:09:12.227802578 +0000 UTC m=+5.968854793 container cleanup 345a6ce2c7c339d4812ff990f8f94c3ec5b5ad8a68105243c1aa74c4cf449093 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-05-06T14:09:12.295 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:12 smithi170 bash[99624]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4 2024-05-06T14:09:12.546 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:12 smithi170 podman[99641]: 2024-05-06 14:09:12.487779461 +0000 UTC m=+0.741974405 container remove 345a6ce2c7c339d4812ff990f8f94c3ec5b5ad8a68105243c1aa74c4cf449093 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-70ce953, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-05-06T14:09:12.546 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:12 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:12.302+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:12.546 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:12 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:12.302+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:45.633819+0000 front 2024-05-06T14:08:45.633749+0000 (oldest deadline 2024-05-06T14:09:10.933542+0000) 2024-05-06T14:09:12.819 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:12 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:12.552+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:12.819 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:12 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:12.552+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:13.125 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:12 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:09:12.834+0000 7f34b81ab700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.210053+0000 front 2024-05-06T14:08:39.209958+0000 (oldest deadline 2024-05-06T14:09:04.509987+0000) 2024-05-06T14:09:13.125 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:12 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:09:12.834+0000 7f34b81ab700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:44.510739+0000 front 2024-05-06T14:08:44.510642+0000 (oldest deadline 2024-05-06T14:09:12.110945+0000) 2024-05-06T14:09:13.126 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:12 smithi170 podman[99730]: 2024-05-06 14:09:12.760573857 +0000 UTC m=+0.018236277 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:09:13.126 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:12 smithi170 podman[99730]: 2024-05-06 14:09:12.863651258 +0000 UTC m=+0.121313681 container create 66f04dadfbd362ba5c9bfa64b63cc1639401942b92bb048c877bae3ddcb046ec (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, RELEASE=squid-70ce953) 2024-05-06T14:09:13.413 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:13 smithi170 podman[99730]: 2024-05-06 14:09:13.133092708 +0000 UTC m=+0.390755128 container init 66f04dadfbd362ba5c9bfa64b63cc1639401942b92bb048c877bae3ddcb046ec (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T14:09:13.413 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:13 smithi170 podman[99730]: 2024-05-06 14:09:13.136527036 +0000 UTC m=+0.394189462 container start 66f04dadfbd362ba5c9bfa64b63cc1639401942b92bb048c877bae3ddcb046ec (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-05-06T14:09:13.413 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:13 smithi170 podman[99730]: 2024-05-06 14:09:13.233214687 +0000 UTC m=+0.490877117 container attach 66f04dadfbd362ba5c9bfa64b63cc1639401942b92bb048c877bae3ddcb046ec (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-05-06T14:09:13.414 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:13 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:13.270+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:13.414 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:13 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:13.270+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:45.633819+0000 front 2024-05-06T14:08:45.633749+0000 (oldest deadline 2024-05-06T14:09:10.933542+0000) 2024-05-06T14:09:13.788 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:13 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:13.509+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:13.788 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:13 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:13.509+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:13.788 INFO:journalctl@ceph.osd.4.smithi170.stdout:May 06 14:09:13 smithi170 podman[99730]: 2024-05-06 14:09:13.420995898 +0000 UTC m=+0.678658328 container died 66f04dadfbd362ba5c9bfa64b63cc1639401942b92bb048c877bae3ddcb046ec (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=) 2024-05-06T14:09:14.069 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:13 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:09:13.871+0000 7f34b81ab700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.210053+0000 front 2024-05-06T14:08:39.209958+0000 (oldest deadline 2024-05-06T14:09:04.509987+0000) 2024-05-06T14:09:14.069 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:13 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:09:13.871+0000 7f34b81ab700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:44.510739+0000 front 2024-05-06T14:08:44.510642+0000 (oldest deadline 2024-05-06T14:09:12.110945+0000) 2024-05-06T14:09:14.162 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.4.service' 2024-05-06T14:09:14.200 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T14:09:14.200 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-05-06T14:09:14.201 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-05-06T14:09:14.201 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.5 2024-05-06T14:09:14.528 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:14 smithi170 systemd[1]: Stopping Ceph osd.5 for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:09:14.528 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:14 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:14.298+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:14.528 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:14 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:14.298+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:45.633819+0000 front 2024-05-06T14:08:45.633749+0000 (oldest deadline 2024-05-06T14:09:10.933542+0000) 2024-05-06T14:09:14.819 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:14 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:14.534+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:14.819 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:14 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:14.534+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:14.819 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:14 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:09:14.539+0000 7f34bbbb7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:09:14.820 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:14 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:09:14.539+0000 7f34bbbb7700 -1 osd.5 160 *** Got signal Terminated *** 2024-05-06T14:09:14.820 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:14 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:09:14.539+0000 7f34bbbb7700 -1 osd.5 160 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T14:09:15.302 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:14 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:09:14.859+0000 7f34b81ab700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.210053+0000 front 2024-05-06T14:08:39.209958+0000 (oldest deadline 2024-05-06T14:09:04.509987+0000) 2024-05-06T14:09:15.303 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:14 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:09:14.859+0000 7f34b81ab700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:44.510739+0000 front 2024-05-06T14:08:44.510642+0000 (oldest deadline 2024-05-06T14:09:12.110945+0000) 2024-05-06T14:09:15.565 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:15 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:15.310+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:15.565 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:15 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:15.310+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:45.633819+0000 front 2024-05-06T14:08:45.633749+0000 (oldest deadline 2024-05-06T14:09:10.933542+0000) 2024-05-06T14:09:15.819 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:15 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:15.572+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:15.819 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:15 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:15.572+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:16.285 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:15 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:09:15.831+0000 7f34b81ab700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.210053+0000 front 2024-05-06T14:08:39.209958+0000 (oldest deadline 2024-05-06T14:09:04.509987+0000) 2024-05-06T14:09:16.285 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:15 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:09:15.831+0000 7f34b81ab700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:44.510739+0000 front 2024-05-06T14:08:44.510642+0000 (oldest deadline 2024-05-06T14:09:12.110945+0000) 2024-05-06T14:09:16.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:16 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:16.292+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:16.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:16 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:16.292+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:45.633819+0000 front 2024-05-06T14:08:45.633749+0000 (oldest deadline 2024-05-06T14:09:10.933542+0000) 2024-05-06T14:09:16.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:16 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:16.292+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:55.034656+0000 front 2024-05-06T14:08:55.034668+0000 (oldest deadline 2024-05-06T14:09:16.134544+0000) 2024-05-06T14:09:17.069 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:16 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:09:16.788+0000 7f34b81ab700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.210053+0000 front 2024-05-06T14:08:39.209958+0000 (oldest deadline 2024-05-06T14:09:04.509987+0000) 2024-05-06T14:09:17.069 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:16 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:09:16.788+0000 7f34b81ab700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:44.510739+0000 front 2024-05-06T14:08:44.510642+0000 (oldest deadline 2024-05-06T14:09:12.110945+0000) 2024-05-06T14:09:17.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:16 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:16.611+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:17.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:16 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:16.611+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:17.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:16 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:16.611+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:51.570996+0000 front 2024-05-06T14:08:51.570939+0000 (oldest deadline 2024-05-06T14:09:16.270784+0000) 2024-05-06T14:09:17.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:17.299+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:17.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:17.299+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:45.633819+0000 front 2024-05-06T14:08:45.633749+0000 (oldest deadline 2024-05-06T14:09:10.933542+0000) 2024-05-06T14:09:17.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:17.299+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:55.034656+0000 front 2024-05-06T14:08:55.034668+0000 (oldest deadline 2024-05-06T14:09:16.134544+0000) 2024-05-06T14:09:18.069 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:09:17.760+0000 7f34b81ab700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.210053+0000 front 2024-05-06T14:08:39.209958+0000 (oldest deadline 2024-05-06T14:09:04.509987+0000) 2024-05-06T14:09:18.069 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:09:17.760+0000 7f34b81ab700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:44.510739+0000 front 2024-05-06T14:08:44.510642+0000 (oldest deadline 2024-05-06T14:09:12.110945+0000) 2024-05-06T14:09:18.069 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:09:17.760+0000 7f34b81ab700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:52.111541+0000 front 2024-05-06T14:08:52.111484+0000 (oldest deadline 2024-05-06T14:09:17.411348+0000) 2024-05-06T14:09:18.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:17.656+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:18.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:17.656+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:18.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:17 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:17.656+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:51.570996+0000 front 2024-05-06T14:08:51.570939+0000 (oldest deadline 2024-05-06T14:09:16.270784+0000) 2024-05-06T14:09:18.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:18.254+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:18.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:18.254+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:45.633819+0000 front 2024-05-06T14:08:45.633749+0000 (oldest deadline 2024-05-06T14:09:10.933542+0000) 2024-05-06T14:09:18.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:18.254+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:55.034656+0000 front 2024-05-06T14:08:55.034668+0000 (oldest deadline 2024-05-06T14:09:16.134544+0000) 2024-05-06T14:09:19.071 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:09:18.748+0000 7f34b81ab700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.210053+0000 front 2024-05-06T14:08:39.209958+0000 (oldest deadline 2024-05-06T14:09:04.509987+0000) 2024-05-06T14:09:19.071 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:09:18.748+0000 7f34b81ab700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:44.510739+0000 front 2024-05-06T14:08:44.510642+0000 (oldest deadline 2024-05-06T14:09:12.110945+0000) 2024-05-06T14:09:19.071 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5[80634]: 2024-05-06T14:09:18.748+0000 7f34b81ab700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:52.111541+0000 front 2024-05-06T14:08:52.111484+0000 (oldest deadline 2024-05-06T14:09:17.411348+0000) 2024-05-06T14:09:19.072 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:18.704+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:19.072 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:18.704+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:19.072 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:18 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:18.704+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:51.570996+0000 front 2024-05-06T14:08:51.570939+0000 (oldest deadline 2024-05-06T14:09:16.270784+0000) 2024-05-06T14:09:19.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:19 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:19.269+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:19.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:19 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:19.269+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:45.633819+0000 front 2024-05-06T14:08:45.633749+0000 (oldest deadline 2024-05-06T14:09:10.933542+0000) 2024-05-06T14:09:19.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:19 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:19.269+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:55.034656+0000 front 2024-05-06T14:08:55.034668+0000 (oldest deadline 2024-05-06T14:09:16.134544+0000) 2024-05-06T14:09:20.004 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:19 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:19.744+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:20.004 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:19 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:19.744+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:20.004 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:19 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:19.744+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:51.570996+0000 front 2024-05-06T14:08:51.570939+0000 (oldest deadline 2024-05-06T14:09:16.270784+0000) 2024-05-06T14:09:20.276 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:20 smithi170 podman[99845]: 2024-05-06 14:09:20.011714596 +0000 UTC m=+5.582269765 container died fa8ea71417a67467ae2b02853de95b65cc26a7b63a9c2f0981cf160915a336ca (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-06T14:09:20.276 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:20.237+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:20.276 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:20.237+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:45.633819+0000 front 2024-05-06T14:08:45.633749+0000 (oldest deadline 2024-05-06T14:09:10.933542+0000) 2024-05-06T14:09:20.277 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:20.237+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:55.034656+0000 front 2024-05-06T14:08:55.034668+0000 (oldest deadline 2024-05-06T14:09:16.134544+0000) 2024-05-06T14:09:20.569 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:20 smithi170 podman[99845]: 2024-05-06 14:09:20.45763751 +0000 UTC m=+6.028192721 container cleanup fa8ea71417a67467ae2b02853de95b65cc26a7b63a9c2f0981cf160915a336ca (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-70ce953, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-06T14:09:20.569 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:20 smithi170 bash[99845]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5 2024-05-06T14:09:21.069 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:20 smithi170 podman[99863]: 2024-05-06 14:09:20.658929106 +0000 UTC m=+0.640591883 container remove fa8ea71417a67467ae2b02853de95b65cc26a7b63a9c2f0981cf160915a336ca (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5, org.label-schema.build-date=20240430, GIT_CLEAN=True, RELEASE=squid-70ce953, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T14:09:21.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:20.697+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:21.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:20.697+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:21.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:20 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:20.697+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:51.570996+0000 front 2024-05-06T14:08:51.570939+0000 (oldest deadline 2024-05-06T14:09:16.270784+0000) 2024-05-06T14:09:21.379 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:21 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:21.285+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:21.379 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:21 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:21.285+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:45.633819+0000 front 2024-05-06T14:08:45.633749+0000 (oldest deadline 2024-05-06T14:09:10.933542+0000) 2024-05-06T14:09:21.380 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:21 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:21.285+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:55.034656+0000 front 2024-05-06T14:08:55.034668+0000 (oldest deadline 2024-05-06T14:09:16.134544+0000) 2024-05-06T14:09:21.380 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:21 smithi170 podman[99955]: 2024-05-06 14:09:20.982989864 +0000 UTC m=+0.018148531 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:09:21.380 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:21 smithi170 podman[99955]: 2024-05-06 14:09:21.08494054 +0000 UTC m=+0.120099191 container create 93b9e76644325e4665c70459cffbf80be236ec4768c4ae16a8f155d124d53401 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-05-06T14:09:21.380 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:21 smithi170 podman[99955]: 2024-05-06 14:09:21.303790756 +0000 UTC m=+0.338949406 container init 93b9e76644325e4665c70459cffbf80be236ec4768c4ae16a8f155d124d53401 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5-deactivate, RELEASE=squid-70ce953, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-05-06T14:09:21.380 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:21 smithi170 podman[99955]: 2024-05-06 14:09:21.306866649 +0000 UTC m=+0.342025301 container start 93b9e76644325e4665c70459cffbf80be236ec4768c4ae16a8f155d124d53401 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-06T14:09:21.380 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:21 smithi170 podman[99955]: 2024-05-06 14:09:21.387089936 +0000 UTC m=+0.422248587 container attach 93b9e76644325e4665c70459cffbf80be236ec4768c4ae16a8f155d124d53401 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430) 2024-05-06T14:09:21.711 INFO:journalctl@ceph.osd.5.smithi170.stdout:May 06 14:09:21 smithi170 podman[99955]: 2024-05-06 14:09:21.604804549 +0000 UTC m=+0.639963206 container died 93b9e76644325e4665c70459cffbf80be236ec4768c4ae16a8f155d124d53401 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, RELEASE=squid-70ce953, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T14:09:22.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:21 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:21.718+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:22.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:21 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:21.718+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:22.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:21 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:21.718+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:51.570996+0000 front 2024-05-06T14:08:51.570939+0000 (oldest deadline 2024-05-06T14:09:16.270784+0000) 2024-05-06T14:09:22.206 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.5.service' 2024-05-06T14:09:22.244 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T14:09:22.244 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-05-06T14:09:22.244 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-05-06T14:09:22.244 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.6 2024-05-06T14:09:22.568 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:22 smithi170 systemd[1]: Stopping Ceph osd.6 for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:09:22.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:22 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:22.241+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:22.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:22 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:22.241+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:45.633819+0000 front 2024-05-06T14:08:45.633749+0000 (oldest deadline 2024-05-06T14:09:10.933542+0000) 2024-05-06T14:09:22.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:22 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:22.241+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:55.034656+0000 front 2024-05-06T14:08:55.034668+0000 (oldest deadline 2024-05-06T14:09:16.134544+0000) 2024-05-06T14:09:23.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:22 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:22.576+0000 7f09023da700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:09:23.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:22 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:22.576+0000 7f09023da700 -1 osd.6 160 *** Got signal Terminated *** 2024-05-06T14:09:23.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:22 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:22.576+0000 7f09023da700 -1 osd.6 160 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T14:09:23.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:22 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:22.707+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:23.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:22 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:22.707+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:23.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:22 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:22.707+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:51.570996+0000 front 2024-05-06T14:08:51.570939+0000 (oldest deadline 2024-05-06T14:09:16.270784+0000) 2024-05-06T14:09:23.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:23 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:23.234+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:23.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:23 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:23.234+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:45.633819+0000 front 2024-05-06T14:08:45.633749+0000 (oldest deadline 2024-05-06T14:09:10.933542+0000) 2024-05-06T14:09:23.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:23 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:23.234+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:55.034656+0000 front 2024-05-06T14:08:55.034668+0000 (oldest deadline 2024-05-06T14:09:16.134544+0000) 2024-05-06T14:09:24.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:23 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:23.667+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:24.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:23 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:23.667+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:24.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:23 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:23.667+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:51.570996+0000 front 2024-05-06T14:08:51.570939+0000 (oldest deadline 2024-05-06T14:09:16.270784+0000) 2024-05-06T14:09:24.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:24.274+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:24.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:24.274+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:45.633819+0000 front 2024-05-06T14:08:45.633749+0000 (oldest deadline 2024-05-06T14:09:10.933542+0000) 2024-05-06T14:09:24.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:24.274+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:55.034656+0000 front 2024-05-06T14:08:55.034668+0000 (oldest deadline 2024-05-06T14:09:16.134544+0000) 2024-05-06T14:09:25.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:24.643+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:25.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:24.643+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:25.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:24 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:24.643+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:51.570996+0000 front 2024-05-06T14:08:51.570939+0000 (oldest deadline 2024-05-06T14:09:16.270784+0000) 2024-05-06T14:09:25.568 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:25.228+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:25.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:25.228+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:45.633819+0000 front 2024-05-06T14:08:45.633749+0000 (oldest deadline 2024-05-06T14:09:10.933542+0000) 2024-05-06T14:09:25.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:25.228+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:55.034656+0000 front 2024-05-06T14:08:55.034668+0000 (oldest deadline 2024-05-06T14:09:16.134544+0000) 2024-05-06T14:09:26.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:25.640+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:26.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:25.640+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:26.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:25 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:25.640+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:51.570996+0000 front 2024-05-06T14:08:51.570939+0000 (oldest deadline 2024-05-06T14:09:16.270784+0000) 2024-05-06T14:09:26.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:26 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:26.236+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:26.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:26 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:26.236+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:45.633819+0000 front 2024-05-06T14:08:45.633749+0000 (oldest deadline 2024-05-06T14:09:10.933542+0000) 2024-05-06T14:09:26.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:26 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:26.236+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:55.034656+0000 front 2024-05-06T14:08:55.034668+0000 (oldest deadline 2024-05-06T14:09:16.134544+0000) 2024-05-06T14:09:27.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:26 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:26.603+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:27.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:26 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:26.603+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:27.069 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:26 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:26.603+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:51.570996+0000 front 2024-05-06T14:08:51.570939+0000 (oldest deadline 2024-05-06T14:09:16.270784+0000) 2024-05-06T14:09:27.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:27.250+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:27.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:27.250+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:45.633819+0000 front 2024-05-06T14:08:45.633749+0000 (oldest deadline 2024-05-06T14:09:10.933542+0000) 2024-05-06T14:09:27.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:27.250+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:55.034656+0000 front 2024-05-06T14:08:55.034668+0000 (oldest deadline 2024-05-06T14:09:16.134544+0000) 2024-05-06T14:09:28.072 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: *** Caught signal (Segmentation fault) ** 2024-05-06T14:09:28.072 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: in thread 7f08ed9ac700 thread_name:safe_timer 2024-05-06T14:09:28.072 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev) 2024-05-06T14:09:28.072 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0909831d20] 2024-05-06T14:09:28.072 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x563a6e06077a] 2024-05-06T14:09:28.072 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x563a6e0ee318] 2024-05-06T14:09:28.072 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x563a6dcda163] 2024-05-06T14:09:28.072 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x563a6df5f1f1] 2024-05-06T14:09:28.072 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x563a6dc55ed4] 2024-05-06T14:09:28.072 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x563a6dc57f3a] 2024-05-06T14:09:28.072 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x563a6dc59ea3] 2024-05-06T14:09:28.073 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x563a6dbbb3ba] 2024-05-06T14:09:28.073 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 10: (CommonSafeTimer::timer_thread()+0x155) [0x563a6e2100e5] 2024-05-06T14:09:28.073 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 11: (CommonSafeTimerThread::entry()+0x11) [0x563a6e211911] 2024-05-06T14:09:28.073 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f09098271ca] 2024-05-06T14:09:28.073 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 13: clone() 2024-05-06T14:09:28.073 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:27.702+0000 7f08ed9ac700 -1 *** Caught signal (Segmentation fault) ** 2024-05-06T14:09:28.073 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: in thread 7f08ed9ac700 thread_name:safe_timer 2024-05-06T14:09:28.073 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:28.073 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev) 2024-05-06T14:09:28.073 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0909831d20] 2024-05-06T14:09:28.074 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x563a6e06077a] 2024-05-06T14:09:28.074 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x563a6e0ee318] 2024-05-06T14:09:28.074 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x563a6dcda163] 2024-05-06T14:09:28.075 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x563a6df5f1f1] 2024-05-06T14:09:28.075 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x563a6dc55ed4] 2024-05-06T14:09:28.075 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x563a6dc57f3a] 2024-05-06T14:09:28.075 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x563a6dc59ea3] 2024-05-06T14:09:28.075 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x563a6dbbb3ba] 2024-05-06T14:09:28.075 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 10: (CommonSafeTimer::timer_thread()+0x155) [0x563a6e2100e5] 2024-05-06T14:09:28.075 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 11: (CommonSafeTimerThread::entry()+0x11) [0x563a6e211911] 2024-05-06T14:09:28.075 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f09098271ca] 2024-05-06T14:09:28.075 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 13: clone() 2024-05-06T14:09:28.075 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-06T14:09:28.075 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:28.075 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -5476> 2024-05-06T14:09:00.612+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:28.075 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -4755> 2024-05-06T14:09:01.583+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:28.076 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -4683> 2024-05-06T14:09:02.559+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:28.076 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -4590> 2024-05-06T14:09:03.531+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:28.076 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -3898> 2024-05-06T14:09:04.565+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:28.076 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -3857> 2024-05-06T14:09:05.559+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:28.076 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -3799> 2024-05-06T14:09:06.537+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:28.076 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -3692> 2024-05-06T14:09:07.531+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:28.076 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -3606> 2024-05-06T14:09:08.557+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:28.076 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -3060> 2024-05-06T14:09:09.607+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:28.077 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -2943> 2024-05-06T14:09:10.642+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:28.077 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -2506> 2024-05-06T14:09:11.594+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:28.077 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -2505> 2024-05-06T14:09:11.594+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:28.077 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -2402> 2024-05-06T14:09:12.552+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:28.077 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -2401> 2024-05-06T14:09:12.552+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:28.077 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -2349> 2024-05-06T14:09:13.509+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:28.077 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -2348> 2024-05-06T14:09:13.509+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:28.077 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -2230> 2024-05-06T14:09:14.534+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:28.077 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -2229> 2024-05-06T14:09:14.534+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:28.077 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -2179> 2024-05-06T14:09:15.572+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:28.077 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -2178> 2024-05-06T14:09:15.572+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:28.077 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -2137> 2024-05-06T14:09:16.611+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:28.078 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -2136> 2024-05-06T14:09:16.611+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:28.078 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -2135> 2024-05-06T14:09:16.611+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:51.570996+0000 front 2024-05-06T14:08:51.570939+0000 (oldest deadline 2024-05-06T14:09:16.270784+0000) 2024-05-06T14:09:28.078 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -1740> 2024-05-06T14:09:17.656+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:28.078 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -1739> 2024-05-06T14:09:17.656+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:28.078 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -1738> 2024-05-06T14:09:17.656+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:51.570996+0000 front 2024-05-06T14:08:51.570939+0000 (oldest deadline 2024-05-06T14:09:16.270784+0000) 2024-05-06T14:09:28.078 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -1626> 2024-05-06T14:09:18.704+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:28.079 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -1625> 2024-05-06T14:09:18.704+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:28.079 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -1624> 2024-05-06T14:09:18.704+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:51.570996+0000 front 2024-05-06T14:08:51.570939+0000 (oldest deadline 2024-05-06T14:09:16.270784+0000) 2024-05-06T14:09:28.079 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -1229> 2024-05-06T14:09:19.744+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:28.079 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -1228> 2024-05-06T14:09:19.744+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:28.079 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -1227> 2024-05-06T14:09:19.744+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:51.570996+0000 front 2024-05-06T14:08:51.570939+0000 (oldest deadline 2024-05-06T14:09:16.270784+0000) 2024-05-06T14:09:28.079 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -1112> 2024-05-06T14:09:20.697+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:28.079 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -1111> 2024-05-06T14:09:20.697+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:28.079 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -1110> 2024-05-06T14:09:20.697+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:51.570996+0000 front 2024-05-06T14:08:51.570939+0000 (oldest deadline 2024-05-06T14:09:16.270784+0000) 2024-05-06T14:09:28.079 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -1057> 2024-05-06T14:09:21.718+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:28.079 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -1056> 2024-05-06T14:09:21.718+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:28.079 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -1055> 2024-05-06T14:09:21.718+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:51.570996+0000 front 2024-05-06T14:08:51.570939+0000 (oldest deadline 2024-05-06T14:09:16.270784+0000) 2024-05-06T14:09:28.079 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -1007> 2024-05-06T14:09:22.576+0000 7f09023da700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:09:28.080 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -1006> 2024-05-06T14:09:22.576+0000 7f09023da700 -1 osd.6 160 *** Got signal Terminated *** 2024-05-06T14:09:28.080 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -1004> 2024-05-06T14:09:22.576+0000 7f09023da700 -1 osd.6 160 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T14:09:28.080 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -984> 2024-05-06T14:09:22.707+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:28.080 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -983> 2024-05-06T14:09:22.707+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:28.080 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -982> 2024-05-06T14:09:22.707+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:51.570996+0000 front 2024-05-06T14:08:51.570939+0000 (oldest deadline 2024-05-06T14:09:16.270784+0000) 2024-05-06T14:09:28.080 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -905> 2024-05-06T14:09:23.667+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:28.081 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -904> 2024-05-06T14:09:23.667+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:28.081 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -903> 2024-05-06T14:09:23.667+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:51.570996+0000 front 2024-05-06T14:08:51.570939+0000 (oldest deadline 2024-05-06T14:09:16.270784+0000) 2024-05-06T14:09:28.081 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -871> 2024-05-06T14:09:24.643+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:28.081 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -870> 2024-05-06T14:09:24.643+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:28.081 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -869> 2024-05-06T14:09:24.643+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:51.570996+0000 front 2024-05-06T14:08:51.570939+0000 (oldest deadline 2024-05-06T14:09:16.270784+0000) 2024-05-06T14:09:28.081 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -672> 2024-05-06T14:09:25.640+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:28.081 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -671> 2024-05-06T14:09:25.640+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:28.081 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -670> 2024-05-06T14:09:25.640+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:51.570996+0000 front 2024-05-06T14:08:51.570939+0000 (oldest deadline 2024-05-06T14:09:16.270784+0000) 2024-05-06T14:09:28.081 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -578> 2024-05-06T14:09:26.603+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:36.368754+0000 front 2024-05-06T14:08:36.368665+0000 (oldest deadline 2024-05-06T14:09:00.468684+0000) 2024-05-06T14:09:28.081 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -577> 2024-05-06T14:09:26.603+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:47.470270+0000 front 2024-05-06T14:08:47.470357+0000 (oldest deadline 2024-05-06T14:09:11.570322+0000) 2024-05-06T14:09:28.081 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: -576> 2024-05-06T14:09:26.603+0000 7f08fe9ce700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:51.570996+0000 front 2024-05-06T14:08:51.570939+0000 (oldest deadline 2024-05-06T14:09:16.270784+0000) 2024-05-06T14:09:28.081 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 0> 2024-05-06T14:09:27.702+0000 7f08ed9ac700 -1 *** Caught signal (Segmentation fault) ** 2024-05-06T14:09:28.082 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: in thread 7f08ed9ac700 thread_name:safe_timer 2024-05-06T14:09:28.082 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:28.082 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: ceph version 19.0.0-2498-g70ce9539 (70ce9539b62a15b9de429c1f0e91a1e012b099aa) squid (dev) 2024-05-06T14:09:28.082 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0909831d20] 2024-05-06T14:09:28.082 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x563a6e06077a] 2024-05-06T14:09:28.082 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x563a6e0ee318] 2024-05-06T14:09:28.082 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x563a6dcda163] 2024-05-06T14:09:28.082 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x563a6df5f1f1] 2024-05-06T14:09:28.082 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x563a6dc55ed4] 2024-05-06T14:09:28.083 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x563a6dc57f3a] 2024-05-06T14:09:28.083 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x563a6dc59ea3] 2024-05-06T14:09:28.083 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x563a6dbbb3ba] 2024-05-06T14:09:28.083 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 10: (CommonSafeTimer::timer_thread()+0x155) [0x563a6e2100e5] 2024-05-06T14:09:28.083 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 11: (CommonSafeTimerThread::entry()+0x11) [0x563a6e211911] 2024-05-06T14:09:28.083 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f09098271ca] 2024-05-06T14:09:28.083 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 13: clone() 2024-05-06T14:09:28.083 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-06T14:09:28.083 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6[85487]: 2024-05-06T14:09:28.083 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:27 smithi170 podman[100062]: 2024-05-06 14:09:27.783346116 +0000 UTC m=+5.309356287 container died 1fbae25547b949cb42e441ed7b34b1818938d31d511d3de23e03dfb4af83bfd4 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=squid-70ce953, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-05-06T14:09:28.488 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:28 smithi170 podman[100062]: 2024-05-06 14:09:28.310846548 +0000 UTC m=+5.836856716 container cleanup 1fbae25547b949cb42e441ed7b34b1818938d31d511d3de23e03dfb4af83bfd4 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=squid-70ce953, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240430) 2024-05-06T14:09:28.488 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:28 smithi170 bash[100062]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6 2024-05-06T14:09:28.488 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:28.226+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:28.488 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:28.226+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:45.633819+0000 front 2024-05-06T14:08:45.633749+0000 (oldest deadline 2024-05-06T14:09:10.933542+0000) 2024-05-06T14:09:28.488 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:28.226+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:55.034656+0000 front 2024-05-06T14:08:55.034668+0000 (oldest deadline 2024-05-06T14:09:16.134544+0000) 2024-05-06T14:09:28.488 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:28 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:28.226+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6830 osd.3 since back 2024-05-06T14:09:02.035601+0000 front 2024-05-06T14:09:02.035589+0000 (oldest deadline 2024-05-06T14:09:27.935444+0000) 2024-05-06T14:09:28.819 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:28 smithi170 podman[100077]: 2024-05-06 14:09:28.495695312 +0000 UTC m=+0.705934193 container remove 1fbae25547b949cb42e441ed7b34b1818938d31d511d3de23e03dfb4af83bfd4 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-06T14:09:28.819 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:28 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.6.service: Main process exited, code=exited, status=139/n/a 2024-05-06T14:09:29.183 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:28 smithi170 podman[100173]: 2024-05-06 14:09:28.913473525 +0000 UTC m=+0.109847428 container create 9420e6d6351b769f1f6ffd0775a1e963dbf85ce837d76c01be3a2e86a79d7604 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5) 2024-05-06T14:09:29.183 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:28 smithi170 podman[100173]: 2024-05-06 14:09:28.822388556 +0000 UTC m=+0.018762462 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:09:29.183 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:29 smithi170 podman[100173]: 2024-05-06 14:09:29.11584812 +0000 UTC m=+0.312222017 container init 9420e6d6351b769f1f6ffd0775a1e963dbf85ce837d76c01be3a2e86a79d7604 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6-deactivate, ceph=True, RELEASE=squid-70ce953, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-05-06T14:09:29.183 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:29 smithi170 podman[100173]: 2024-05-06 14:09:29.118844288 +0000 UTC m=+0.315218185 container start 9420e6d6351b769f1f6ffd0775a1e963dbf85ce837d76c01be3a2e86a79d7604 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T14:09:29.184 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:29 smithi170 podman[100173]: 2024-05-06 14:09:29.190798148 +0000 UTC m=+0.387172050 container attach 9420e6d6351b769f1f6ffd0775a1e963dbf85ce837d76c01be3a2e86a79d7604 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-70ce953, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-05-06T14:09:29.569 INFO:journalctl@ceph.osd.6.smithi170.stdout:May 06 14:09:29 smithi170 podman[100173]: 2024-05-06 14:09:29.40917333 +0000 UTC m=+0.605547229 container died 9420e6d6351b769f1f6ffd0775a1e963dbf85ce837d76c01be3a2e86a79d7604 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-6-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-70ce953, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-06T14:09:29.570 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:29.197+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:29.570 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:29.197+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:45.633819+0000 front 2024-05-06T14:08:45.633749+0000 (oldest deadline 2024-05-06T14:09:10.933542+0000) 2024-05-06T14:09:29.570 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:29.197+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:55.034656+0000 front 2024-05-06T14:08:55.034668+0000 (oldest deadline 2024-05-06T14:09:16.134544+0000) 2024-05-06T14:09:29.570 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:29 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:29.197+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6830 osd.3 since back 2024-05-06T14:09:02.035601+0000 front 2024-05-06T14:09:02.035589+0000 (oldest deadline 2024-05-06T14:09:27.935444+0000) 2024-05-06T14:09:30.055 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.6.service' 2024-05-06T14:09:30.098 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T14:09:30.098 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-05-06T14:09:30.098 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-05-06T14:09:30.099 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.7 2024-05-06T14:09:30.321 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:30 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:30.165+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:30.321 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:30 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:30.165+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:45.633819+0000 front 2024-05-06T14:08:45.633749+0000 (oldest deadline 2024-05-06T14:09:10.933542+0000) 2024-05-06T14:09:30.321 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:30 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:30.165+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:55.034656+0000 front 2024-05-06T14:08:55.034668+0000 (oldest deadline 2024-05-06T14:09:16.134544+0000) 2024-05-06T14:09:30.322 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:30 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:30.165+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6830 osd.3 since back 2024-05-06T14:09:02.035601+0000 front 2024-05-06T14:09:02.035589+0000 (oldest deadline 2024-05-06T14:09:27.935444+0000) 2024-05-06T14:09:30.322 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:30 smithi170 systemd[1]: Stopping Ceph osd.7 for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:09:30.818 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:30 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:30.446+0000 7f3ad60f8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-06T14:09:30.819 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:30 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:30.446+0000 7f3ad60f8700 -1 osd.7 160 *** Got signal Terminated *** 2024-05-06T14:09:30.819 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:30 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:30.446+0000 7f3ad60f8700 -1 osd.7 160 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-06T14:09:31.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:31 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:31.177+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:31.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:31 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:31.177+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:45.633819+0000 front 2024-05-06T14:08:45.633749+0000 (oldest deadline 2024-05-06T14:09:10.933542+0000) 2024-05-06T14:09:31.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:31 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:31.177+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:55.034656+0000 front 2024-05-06T14:08:55.034668+0000 (oldest deadline 2024-05-06T14:09:16.134544+0000) 2024-05-06T14:09:31.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:31 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:31.177+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6830 osd.3 since back 2024-05-06T14:09:02.035601+0000 front 2024-05-06T14:09:02.035589+0000 (oldest deadline 2024-05-06T14:09:27.935444+0000) 2024-05-06T14:09:32.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:32 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:32.198+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:32.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:32 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:32.198+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:45.633819+0000 front 2024-05-06T14:08:45.633749+0000 (oldest deadline 2024-05-06T14:09:10.933542+0000) 2024-05-06T14:09:32.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:32 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:32.198+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:55.034656+0000 front 2024-05-06T14:08:55.034668+0000 (oldest deadline 2024-05-06T14:09:16.134544+0000) 2024-05-06T14:09:32.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:32 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:32.198+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6830 osd.3 since back 2024-05-06T14:09:02.035601+0000 front 2024-05-06T14:09:02.035589+0000 (oldest deadline 2024-05-06T14:09:27.935444+0000) 2024-05-06T14:09:33.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:33 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:33.241+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:33.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:33 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:33.241+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:45.633819+0000 front 2024-05-06T14:08:45.633749+0000 (oldest deadline 2024-05-06T14:09:10.933542+0000) 2024-05-06T14:09:33.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:33 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:33.241+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:55.034656+0000 front 2024-05-06T14:08:55.034668+0000 (oldest deadline 2024-05-06T14:09:16.134544+0000) 2024-05-06T14:09:33.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:33 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:33.241+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6830 osd.3 since back 2024-05-06T14:09:02.035601+0000 front 2024-05-06T14:09:02.035589+0000 (oldest deadline 2024-05-06T14:09:27.935444+0000) 2024-05-06T14:09:34.568 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:34 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:34.222+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:34.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:34 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:34.222+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:45.633819+0000 front 2024-05-06T14:08:45.633749+0000 (oldest deadline 2024-05-06T14:09:10.933542+0000) 2024-05-06T14:09:34.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:34 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:34.222+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:55.034656+0000 front 2024-05-06T14:08:55.034668+0000 (oldest deadline 2024-05-06T14:09:16.134544+0000) 2024-05-06T14:09:34.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:34 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:34.222+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6830 osd.3 since back 2024-05-06T14:09:02.035601+0000 front 2024-05-06T14:09:02.035589+0000 (oldest deadline 2024-05-06T14:09:27.935444+0000) 2024-05-06T14:09:34.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:34 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:34.222+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.170:6804 osd.4 since back 2024-05-06T14:09:07.935820+0000 front 2024-05-06T14:09:07.935889+0000 (oldest deadline 2024-05-06T14:09:33.835762+0000) 2024-05-06T14:09:35.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:35.221+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6806 osd.0 since back 2024-05-06T14:08:39.733368+0000 front 2024-05-06T14:08:39.733451+0000 (oldest deadline 2024-05-06T14:09:05.633138+0000) 2024-05-06T14:09:35.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:35.221+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6814 osd.1 since back 2024-05-06T14:08:45.633819+0000 front 2024-05-06T14:08:45.633749+0000 (oldest deadline 2024-05-06T14:09:10.933542+0000) 2024-05-06T14:09:35.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:35.221+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6822 osd.2 since back 2024-05-06T14:08:55.034656+0000 front 2024-05-06T14:08:55.034668+0000 (oldest deadline 2024-05-06T14:09:16.134544+0000) 2024-05-06T14:09:35.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:35.221+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.77:6830 osd.3 since back 2024-05-06T14:09:02.035601+0000 front 2024-05-06T14:09:02.035589+0000 (oldest deadline 2024-05-06T14:09:27.935444+0000) 2024-05-06T14:09:35.569 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:35 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7[90454]: 2024-05-06T14:09:35.221+0000 7f3ace6e4700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.170:6804 osd.4 since back 2024-05-06T14:09:07.935820+0000 front 2024-05-06T14:09:07.935889+0000 (oldest deadline 2024-05-06T14:09:33.835762+0000) 2024-05-06T14:09:36.024 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:35 smithi170 podman[100282]: 2024-05-06 14:09:35.780727415 +0000 UTC m=+5.439586836 container died 82e3dd2bbdd97faf327b4b2d80c22ffbb9e3d4e244d9fa234a57643c481a375a (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-70ce953, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-06T14:09:36.299 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:36 smithi170 podman[100282]: 2024-05-06 14:09:36.122072875 +0000 UTC m=+5.780932338 container cleanup 82e3dd2bbdd97faf327b4b2d80c22ffbb9e3d4e244d9fa234a57643c481a375a (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-70ce953, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-05-06T14:09:36.299 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:36 smithi170 bash[100282]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7 2024-05-06T14:09:36.568 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:36 smithi170 podman[100300]: 2024-05-06 14:09:36.306757969 +0000 UTC m=+0.525940349 container remove 82e3dd2bbdd97faf327b4b2d80c22ffbb9e3d4e244d9fa234a57643c481a375a (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-70ce953, CEPH_POINT_RELEASE=) 2024-05-06T14:09:37.044 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:36 smithi170 podman[100393]: 2024-05-06 14:09:36.649089766 +0000 UTC m=+0.018186087 image pull e4ce8ad5d0c4b9ea60bc5f6b8b0fdb5efbe59bf7771fb0920fc2200e6ae546b0 quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa 2024-05-06T14:09:37.044 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:36 smithi170 podman[100393]: 2024-05-06 14:09:36.749161152 +0000 UTC m=+0.118257472 container create 392f0dd96e118170f599923663666b7a1b9f820bb5f2efd7b378665266415472 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-70ce953, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, GIT_CLEAN=True) 2024-05-06T14:09:37.044 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:37 smithi170 podman[100393]: 2024-05-06 14:09:37.052505939 +0000 UTC m=+0.421602259 container init 392f0dd96e118170f599923663666b7a1b9f820bb5f2efd7b378665266415472 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-70ce953, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-06T14:09:37.319 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:37 smithi170 podman[100393]: 2024-05-06 14:09:37.055564914 +0000 UTC m=+0.424661242 container start 392f0dd96e118170f599923663666b7a1b9f820bb5f2efd7b378665266415472 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, ceph=True) 2024-05-06T14:09:37.319 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:37 smithi170 podman[100393]: 2024-05-06 14:09:37.134962195 +0000 UTC m=+0.504058521 container attach 392f0dd96e118170f599923663666b7a1b9f820bb5f2efd7b378665266415472 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-70ce953) 2024-05-06T14:09:37.629 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:37 smithi170 podman[100393]: 2024-05-06 14:09:37.341728772 +0000 UTC m=+0.710825085 container died 392f0dd96e118170f599923663666b7a1b9f820bb5f2efd7b378665266415472 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, RELEASE=squid-70ce953, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-05-06T14:09:37.880 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.7.service' 2024-05-06T14:09:37.901 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:37 smithi170 podman[100419]: 2024-05-06 14:09:37.871811924 +0000 UTC m=+0.520595624 container remove 392f0dd96e118170f599923663666b7a1b9f820bb5f2efd7b378665266415472 (image=quay.ceph.io/ceph-ci/ceph:70ce9539b62a15b9de429c1f0e91a1e012b099aa, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-70ce953, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=) 2024-05-06T14:09:37.901 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:37 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.7.service: Deactivated successfully. 2024-05-06T14:09:37.902 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:37 smithi170 systemd[1]: Stopped Ceph osd.7 for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T14:09:37.902 INFO:journalctl@ceph.osd.7.smithi170.stdout:May 06 14:09:37 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@osd.7.service: Consumed 4.358s CPU time. 2024-05-06T14:09:37.916 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T14:09:37.916 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-05-06T14:09:37.916 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-05-06T14:09:37.917 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@prometheus.a 2024-05-06T14:09:38.241 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 14:09:38 smithi170 systemd[1]: Stopping Ceph prometheus.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:09:38.517 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 14:09:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T14:09:38.249Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-05-06T14:09:38.518 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 14:09:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T14:09:38.249Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-05-06T14:09:38.518 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 14:09:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T14:09:38.249Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-05-06T14:09:38.518 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 14:09:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T14:09:38.249Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-05-06T14:09:38.518 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 14:09:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T14:09:38.249Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-05-06T14:09:38.518 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 14:09:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T14:09:38.249Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-05-06T14:09:38.518 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 14:09:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T14:09:38.249Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-05-06T14:09:38.518 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 14:09:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T14:09:38.249Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-05-06T14:09:38.518 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 14:09:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T14:09:38.249Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-05-06T14:09:38.518 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 14:09:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T14:09:38.250Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-05-06T14:09:38.518 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 14:09:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T14:09:38.250Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-05-06T14:09:38.518 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 14:09:38 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a[62029]: ts=2024-05-06T14:09:38.250Z caller=main.go:1147 level=info msg="See you next time!" 2024-05-06T14:09:38.518 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 14:09:38 smithi170 podman[100498]: 2024-05-06 14:09:38.259813777 +0000 UTC m=+0.114537841 container died 10d5c513bb9b635f812a95f30f073ab5eeb2b6d805f04f8ab31fafd543a5d7c4 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T14:09:38.818 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 14:09:38 smithi170 podman[100498]: 2024-05-06 14:09:38.649668317 +0000 UTC m=+0.504392423 container cleanup 10d5c513bb9b635f812a95f30f073ab5eeb2b6d805f04f8ab31fafd543a5d7c4 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-06T14:09:38.818 INFO:journalctl@ceph.prometheus.a.smithi170.stdout:May 06 14:09:38 smithi170 bash[100498]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-prometheus-a 2024-05-06T14:09:39.002 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@prometheus.a.service' 2024-05-06T14:09:39.035 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-06T14:09:39.035 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-05-06T14:09:39.035 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 --force --keep-logs 2024-05-06T14:09:40.794 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 14:09:40 smithi077 systemd[1]: Stopping Ceph node-exporter.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:09:41.067 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 14:09:40 smithi077 podman[132594]: 2024-05-06 14:09:40.849745323 +0000 UTC m=+0.113906908 container died 67f45fa4968b50f9937d0731b271a5cef5e060f0f9563e0ac5a542212f6adbcf (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T14:09:41.368 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 14:09:41 smithi077 podman[132594]: 2024-05-06 14:09:41.174744439 +0000 UTC m=+0.438906070 container cleanup 67f45fa4968b50f9937d0731b271a5cef5e060f0f9563e0ac5a542212f6adbcf (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T14:09:41.368 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 14:09:41 smithi077 bash[132594]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a 2024-05-06T14:09:41.766 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 14:09:41 smithi077 podman[132609]: 2024-05-06 14:09:41.375803946 +0000 UTC m=+0.524666667 container remove 67f45fa4968b50f9937d0731b271a5cef5e060f0f9563e0ac5a542212f6adbcf (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-06T14:09:41.766 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 14:09:41 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-05-06T14:09:41.766 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 14:09:41 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-05-06T14:09:41.766 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 14:09:41 smithi077 systemd[1]: Stopped Ceph node-exporter.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T14:09:41.767 INFO:journalctl@ceph.node-exporter.a.smithi077.stdout:May 06 14:09:41 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@node-exporter.a.service: Consumed 3.281s CPU time. 2024-05-06T14:09:42.045 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 14:09:41 smithi077 systemd[1]: Stopping Ceph alertmanager.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:09:42.045 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 14:09:42 smithi077 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a[79442]: ts=2024-05-06T14:09:42.020Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-05-06T14:09:42.347 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 14:09:42 smithi077 podman[132715]: 2024-05-06 14:09:42.061748822 +0000 UTC m=+0.150645908 container died 997175e3e64efa0dd530a32eda4af713c04a08ecf93df0f3ab8d801e0741ab5e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T14:09:42.664 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 14:09:42 smithi077 podman[132715]: 2024-05-06 14:09:42.44594689 +0000 UTC m=+0.534843979 container cleanup 997175e3e64efa0dd530a32eda4af713c04a08ecf93df0f3ab8d801e0741ab5e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T14:09:42.664 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 14:09:42 smithi077 bash[132715]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a 2024-05-06T14:09:42.926 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 14:09:42 smithi077 podman[132730]: 2024-05-06 14:09:42.672034179 +0000 UTC m=+0.601624419 container remove 997175e3e64efa0dd530a32eda4af713c04a08ecf93df0f3ab8d801e0741ab5e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-06T14:09:42.926 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 14:09:42 smithi077 podman[132730]: 2024-05-06 14:09:42.747960051 +0000 UTC m=+0.677550282 volume remove 96fe4f4a20341b2da21ba624b7eec8d638433ed0e509b5d2c29d85764d10cd33 2024-05-06T14:09:42.926 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 14:09:42 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-05-06T14:09:42.927 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 14:09:42 smithi077 systemd[1]: Stopped Ceph alertmanager.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T14:09:42.927 INFO:journalctl@ceph.alertmanager.a.smithi077.stdout:May 06 14:09:42 smithi077 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@alertmanager.a.service: Consumed 1.525s CPU time. 2024-05-06T14:10:05.911 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 --force --keep-logs 2024-05-06T14:10:07.562 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 14:10:07 smithi170 systemd[1]: Stopping Ceph node-exporter.b for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:10:07.818 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 14:10:07 smithi170 podman[100896]: 2024-05-06 14:10:07.56965168 +0000 UTC m=+0.119205509 container died 02081b71a5dd107771aa79ad7d5f31bc55e971aac33ebf9027b7380d044bfd77 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T14:10:08.185 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 14:10:07 smithi170 podman[100896]: 2024-05-06 14:10:07.909238811 +0000 UTC m=+0.458792637 container cleanup 02081b71a5dd107771aa79ad7d5f31bc55e971aac33ebf9027b7380d044bfd77 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T14:10:08.185 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 14:10:07 smithi170 bash[100896]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b 2024-05-06T14:10:08.518 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 14:10:08 smithi170 podman[100910]: 2024-05-06 14:10:08.193674209 +0000 UTC m=+0.628484719 container remove 02081b71a5dd107771aa79ad7d5f31bc55e971aac33ebf9027b7380d044bfd77 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-06T14:10:08.518 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 14:10:08 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-05-06T14:10:08.518 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 14:10:08 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-05-06T14:10:08.518 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 14:10:08 smithi170 systemd[1]: Stopped Ceph node-exporter.b for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T14:10:08.518 INFO:journalctl@ceph.node-exporter.b.smithi170.stdout:May 06 14:10:08 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@node-exporter.b.service: Consumed 3.301s CPU time. 2024-05-06T14:10:08.805 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 14:10:08 smithi170 systemd[1]: Stopping Ceph grafana.a for 20e59cb4-0bac-11ef-bc97-c7b262605968... 2024-05-06T14:10:09.064 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 14:10:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=server t=2024-05-06T14:10:08.813535811Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-05-06T14:10:09.064 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 14:10:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=tracing t=2024-05-06T14:10:08.813875394Z level=info msg="Closing tracing" 2024-05-06T14:10:09.065 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 14:10:08 smithi170 ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a[64248]: logger=ticker t=2024-05-06T14:10:08.813915485Z level=info msg=stopped last_tick=2024-05-06T14:10:00Z 2024-05-06T14:10:09.065 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 14:10:08 smithi170 podman[101020]: 2024-05-06 14:10:08.823780376 +0000 UTC m=+0.136868566 container died 140bb9a2760a2fa6f9beba2b09b5ac6feda1b11d3acb11aaf35ee8a48a326501 (image=quay.io/ceph/grafana:9.4.12, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-05-06T14:10:09.318 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 14:10:09 smithi170 podman[101020]: 2024-05-06 14:10:09.154990763 +0000 UTC m=+0.468078948 container cleanup 140bb9a2760a2fa6f9beba2b09b5ac6feda1b11d3acb11aaf35ee8a48a326501 (image=quay.io/ceph/grafana:9.4.12, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-05-06T14:10:09.318 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 14:10:09 smithi170 bash[101020]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a 2024-05-06T14:10:09.591 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 14:10:09 smithi170 podman[101038]: 2024-05-06 14:10:09.339416905 +0000 UTC m=+0.517063898 container remove 140bb9a2760a2fa6f9beba2b09b5ac6feda1b11d3acb11aaf35ee8a48a326501 (image=quay.io/ceph/grafana:9.4.12, name=ceph-20e59cb4-0bac-11ef-bc97-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-05-06T14:10:09.592 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 14:10:09 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@grafana.a.service: Deactivated successfully. 2024-05-06T14:10:09.592 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 14:10:09 smithi170 systemd[1]: Stopped Ceph grafana.a for 20e59cb4-0bac-11ef-bc97-c7b262605968. 2024-05-06T14:10:09.592 INFO:journalctl@ceph.grafana.a.smithi170.stdout:May 06 14:10:09 smithi170 systemd[1]: ceph-20e59cb4-0bac-11ef-bc97-c7b262605968@grafana.a.service: Consumed 11.530s CPU time. 2024-05-06T14:10:21.213 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-06T14:10:21.246 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-06T14:10:21.273 INFO:tasks.cephadm:Archiving crash dumps... 2024-05-06T14:10:21.275 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688245/remote/smithi077/crash 2024-05-06T14:10:21.276 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/crash -- . 2024-05-06T14:10:21.329 INFO:teuthology.orchestra.run.smithi077.stderr:tar: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/crash: Cannot open: No such file or directory 2024-05-06T14:10:21.330 INFO:teuthology.orchestra.run.smithi077.stderr:tar: Error is not recoverable: exiting now 2024-05-06T14:10:21.331 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688245/remote/smithi170/crash 2024-05-06T14:10:21.332 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/crash -- . 2024-05-06T14:10:21.363 INFO:teuthology.orchestra.run.smithi170.stderr:tar: /var/lib/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/crash: Cannot open: No such file or directory 2024-05-06T14:10:21.363 INFO:teuthology.orchestra.run.smithi170.stderr:tar: Error is not recoverable: exiting now 2024-05-06T14:10:21.364 INFO:tasks.cephadm:Checking cluster log for badness... 2024-05-06T14:10:21.365 DEBUG:teuthology.orchestra.run.smithi077:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-05-06T14:10:21.403 INFO:teuthology.orchestra.run.smithi077.stdout:2024-05-06T13:44:23.366285+0000 mon.a (mon.0) 996 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-06T14:10:21.403 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-05-06T14:10:21.403 DEBUG:teuthology.orchestra.run.smithi077:> sudo egrep '\[SEC\]' /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-05-06T14:10:21.479 DEBUG:teuthology.orchestra.run.smithi077:> sudo egrep '\[ERR\]' /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-05-06T14:10:21.549 DEBUG:teuthology.orchestra.run.smithi077:> sudo egrep '\[WRN\]' /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-05-06T14:10:21.622 INFO:teuthology.orchestra.run.smithi077.stdout:2024-05-06T13:44:23.366285+0000 mon.a (mon.0) 996 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-06T14:10:21.622 INFO:tasks.cephadm:Compressing logs... 2024-05-06T14:10:21.623 DEBUG:teuthology.orchestra.run.smithi077:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-06T14:10:21.667 DEBUG:teuthology.orchestra.run.smithi170:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-06T14:10:21.690 INFO:teuthology.orchestra.run.smithi077.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph.log 2024-05-06T14:10:21.690 INFO:teuthology.orchestra.run.smithi077.stderr:: No such file or directory 2024-05-06T14:10:21.690 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-osd.2.log 2024-05-06T14:10:21.690 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-client.rgw.foo.smithi077.xctpgy.log 2024-05-06T14:10:21.690 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph.audit.log 2024-05-06T14:10:21.695 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-client.rgw.foo.smithi077.xctpgy.log: gzip -5 --verbose -- /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-osd.3.log 2024-05-06T14:10:21.695 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-osd.1.log 2024-05-06T14:10:21.695 INFO:teuthology.orchestra.run.smithi077.stderr: 76.5% -- replaced with /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-client.rgw.foo.smithi077.xctpgy.log.gz 2024-05-06T14:10:21.695 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-mgr.y.log 2024-05-06T14:10:21.695 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-osd.0.log 2024-05-06T14:10:21.695 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-mon.c.log 2024-05-06T14:10:21.697 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-osd.0.log: 93.9% -- replaced with /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph.log.gz 2024-05-06T14:10:21.701 INFO:teuthology.orchestra.run.smithi077.stderr: 94.4% -- replaced with /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph.audit.log.gz 2024-05-06T14:10:21.707 INFO:teuthology.orchestra.run.smithi170.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-client.rgw.foo.smithi170.rackmk.log 2024-05-06T14:10:21.707 INFO:teuthology.orchestra.run.smithi170.stderr:: No such file or directory 2024-05-06T14:10:21.707 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph.log 2024-05-06T14:10:21.707 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-client.rgw.foo.smithi170.rackmk.log: gzip -5 --verbose -- /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph.audit.log 2024-05-06T14:10:21.707 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-osd.7.log 2024-05-06T14:10:21.708 INFO:teuthology.orchestra.run.smithi170.stderr: 76.5% -- replaced with /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-client.rgw.foo.smithi170.rackmk.log.gz 2024-05-06T14:10:21.708 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-osd.5.log 2024-05-06T14:10:21.708 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-osd.4.log 2024-05-06T14:10:21.708 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-mgr.x.log 2024-05-06T14:10:21.708 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-osd.6.log 2024-05-06T14:10:21.708 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph.cephadm.log 2024-05-06T14:10:21.708 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-volume.log 2024-05-06T14:10:21.708 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph.cephadm.log: 89.0% -- replaced with /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph.log.gz 2024-05-06T14:10:21.709 INFO:teuthology.orchestra.run.smithi170.stderr: 91.7% -- replaced with /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph.cephadm.log.gzgzip -5 --verbose -- /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-mon.b.log 2024-05-06T14:10:21.709 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-05-06T14:10:21.709 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-06T14:10:21.709 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-mon.b.log: 91.0% -- replaced with /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph.audit.log.gz 2024-05-06T14:10:21.710 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-mon.a.log 2024-05-06T14:10:21.711 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph.cephadm.log 2024-05-06T14:10:21.714 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cephadm.log: 91.0% -- replaced with /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-volume.log.gz 2024-05-06T14:10:21.715 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/tcmu-runner.log 2024-05-06T14:10:21.718 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph.cephadm.log: 94.9% -- replaced with /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph.cephadm.log.gz 2024-05-06T14:10:21.721 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-volume.log 2024-05-06T14:10:21.723 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/tcmu-runner.log: 90.2% -- replaced with /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/tcmu-runner.log.gz 2024-05-06T14:10:21.733 INFO:teuthology.orchestra.run.smithi170.stderr: 90.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-06T14:10:21.734 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-06T14:10:21.766 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-volume.log: /var/log/ceph/cephadm.log: 91.0% -- replaced with /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-volume.log.gz 2024-05-06T14:10:21.791 INFO:teuthology.orchestra.run.smithi077.stderr: 92.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-06T14:10:21.813 INFO:teuthology.orchestra.run.smithi170.stderr: 89.7% -- replaced with /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-mgr.x.log.gz 2024-05-06T14:10:22.216 INFO:teuthology.orchestra.run.smithi077.stderr: 89.1% -- replaced with /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-mgr.y.log.gz 2024-05-06T14:10:22.248 INFO:teuthology.orchestra.run.smithi170.stderr: 92.5% -- replaced with /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-mon.b.log.gz 2024-05-06T14:10:22.471 INFO:teuthology.orchestra.run.smithi077.stderr: 92.5% -- replaced with /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-mon.c.log.gz 2024-05-06T14:10:23.394 INFO:teuthology.orchestra.run.smithi170.stderr: 93.2% -- replaced with /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-osd.6.log.gz 2024-05-06T14:10:23.472 INFO:teuthology.orchestra.run.smithi077.stderr: 93.3% -- replaced with /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-osd.2.log.gz 2024-05-06T14:10:23.552 INFO:teuthology.orchestra.run.smithi077.stderr: 91.0% -- replaced with /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-mon.a.log.gz 2024-05-06T14:10:23.570 INFO:teuthology.orchestra.run.smithi170.stderr: 93.4% -- replaced with /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-osd.5.log.gz 2024-05-06T14:10:23.745 INFO:teuthology.orchestra.run.smithi170.stderr: 93.9% -- replaced with /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-osd.7.log.gz 2024-05-06T14:10:23.808 INFO:teuthology.orchestra.run.smithi170.stderr: 93.5% -- replaced with /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-osd.4.log.gz 2024-05-06T14:10:23.809 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-05-06T14:10:23.809 INFO:teuthology.orchestra.run.smithi170.stderr:real 0m2.126s 2024-05-06T14:10:23.809 INFO:teuthology.orchestra.run.smithi170.stderr:user 0m8.148s 2024-05-06T14:10:23.810 INFO:teuthology.orchestra.run.smithi170.stderr:sys 0m0.330s 2024-05-06T14:10:23.869 INFO:teuthology.orchestra.run.smithi077.stderr: 93.4% -- replaced with /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-osd.1.log.gz 2024-05-06T14:10:24.066 INFO:teuthology.orchestra.run.smithi077.stderr: 93.4% -- replaced with /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-osd.0.log.gz 2024-05-06T14:10:24.196 INFO:teuthology.orchestra.run.smithi077.stderr: 93.4% -- replaced with /var/log/ceph/20e59cb4-0bac-11ef-bc97-c7b262605968/ceph-osd.3.log.gz 2024-05-06T14:10:24.197 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-05-06T14:10:24.198 INFO:teuthology.orchestra.run.smithi077.stderr:real 0m2.518s 2024-05-06T14:10:24.198 INFO:teuthology.orchestra.run.smithi077.stderr:user 0m11.585s 2024-05-06T14:10:24.198 INFO:teuthology.orchestra.run.smithi077.stderr:sys 0m0.403s 2024-05-06T14:10:24.198 INFO:tasks.cephadm:Archiving logs... 2024-05-06T14:10:24.198 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688245/remote/smithi077/log 2024-05-06T14:10:24.199 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-06T14:10:25.703 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688245/remote/smithi170/log 2024-05-06T14:10:25.704 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-06T14:10:26.980 INFO:tasks.cephadm:Removing cluster... 2024-05-06T14:10:26.980 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 --force 2024-05-06T14:10:27.273 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 20e59cb4-0bac-11ef-bc97-c7b262605968 --force 2024-05-06T14:10:27.564 INFO:tasks.cephadm:Removing cephadm ... 2024-05-06T14:10:27.565 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-06T14:10:27.579 DEBUG:teuthology.orchestra.run.smithi170:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-06T14:10:27.594 INFO:tasks.cephadm:Teardown complete 2024-05-06T14:10:27.595 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-06T14:10:27.605 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-06T14:10:27.606 DEBUG:teuthology.orchestra.run.smithi077:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-06T14:10:27.622 DEBUG:teuthology.orchestra.run.smithi170:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-06T14:10:27.633 INFO:teuthology.orchestra.run.smithi077.stderr:bash: line 1: ntpq: command not found 2024-05-06T14:10:27.638 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-06T14:10:27.638 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-05-06T14:10:27.638 INFO:teuthology.orchestra.run.smithi077.stdout:^* hv01.front.sepia.ceph.com 3 6 377 31 +1095us[+1057us] +/- 61ms 2024-05-06T14:10:27.638 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 30 -1538us[-1538us] +/- 51ms 2024-05-06T14:10:27.638 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 +1338us[+1338us] +/- 110ms 2024-05-06T14:10:27.638 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T14:10:27.647 INFO:teuthology.orchestra.run.smithi170.stderr:bash: line 1: ntpq: command not found 2024-05-06T14:10:27.652 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-06T14:10:27.652 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2024-05-06T14:10:27.652 INFO:teuthology.orchestra.run.smithi170.stdout:^* hv01.front.sepia.ceph.com 3 6 377 31 +1079us[+1058us] +/- 61ms 2024-05-06T14:10:27.653 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 33 -1493us[-1515us] +/- 51ms 2024-05-06T14:10:27.653 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 +1335us[+1313us] +/- 110ms 2024-05-06T14:10:27.653 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-06T14:10:27.653 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-06T14:10:27.664 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-06T14:10:27.665 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-06T14:10:27.675 DEBUG:teuthology.orchestra.run.smithi077:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-06T14:10:27.763 DEBUG:teuthology.orchestra.run.smithi170:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-06T14:10:27.836 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-06T14:10:27.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T14:10:27.864 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-06T14:10:27.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T14:10:27.891 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-06T14:10:27.903 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-06T14:10:27.913 INFO:teuthology.task.internal:Duration was 2920.546448 seconds 2024-05-06T14:10:27.913 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-06T14:10:27.923 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-06T14:10:27.923 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-06T14:10:27.926 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-06T14:10:27.960 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-06T14:10:27.967 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-06T14:10:28.358 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-06T14:10:28.358 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi077.front.sepia.ceph.com 2024-05-06T14:10:28.358 DEBUG:teuthology.orchestra.run.smithi077:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-06T14:10:28.387 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi170.front.sepia.ceph.com 2024-05-06T14:10:28.388 DEBUG:teuthology.orchestra.run.smithi170:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-06T14:10:28.423 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-06T14:10:28.423 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-06T14:10:28.432 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-06T14:10:28.489 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-06T14:10:28.489 DEBUG:teuthology.orchestra.run.smithi077:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-06T14:10:28.497 DEBUG:teuthology.orchestra.run.smithi170:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-06T14:10:29.197 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-06T14:10:29.206 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-06T14:10:29.206 DEBUG:teuthology.orchestra.run.smithi077:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-06T14:10:29.230 DEBUG:teuthology.orchestra.run.smithi170:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-06T14:10:29.255 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-06T14:10:29.266 DEBUG:teuthology.orchestra.run.smithi077:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-06T14:10:29.273 DEBUG:teuthology.orchestra.run.smithi170:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-06T14:10:29.292 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = core 2024-05-06T14:10:29.317 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = core 2024-05-06T14:10:29.338 DEBUG:teuthology.orchestra.run.smithi077:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-06T14:10:29.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T14:10:29.360 DEBUG:teuthology.orchestra.run.smithi170:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-06T14:10:29.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-06T14:10:29.392 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-06T14:10:29.403 INFO:teuthology.task.internal:Transferring archived files... 2024-05-06T14:10:29.403 DEBUG:teuthology.misc:Transferring archived files from smithi077:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688245/remote/smithi077 2024-05-06T14:10:29.404 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-06T14:10:29.455 DEBUG:teuthology.misc:Transferring archived files from smithi170:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688245/remote/smithi170 2024-05-06T14:10:29.455 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-06T14:10:29.495 INFO:teuthology.task.internal:Removing archive directory... 2024-05-06T14:10:29.495 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-06T14:10:29.498 DEBUG:teuthology.orchestra.run.smithi170:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-06T14:10:29.542 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-06T14:10:29.552 INFO:teuthology.task.internal:Not uploading archives. 2024-05-06T14:10:29.553 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-06T14:10:29.565 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-06T14:10:29.565 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-06T14:10:29.567 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-06T14:10:29.579 INFO:teuthology.orchestra.run.smithi077.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 6 14:10 /home/ubuntu/cephtest 2024-05-06T14:10:29.596 INFO:teuthology.orchestra.run.smithi170.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 6 14:10 /home/ubuntu/cephtest 2024-05-06T14:10:29.597 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-06T14:10:29.607 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-06T14:10:29.629 INFO:teuthology.nuke:Checking targets against current locks 2024-05-06T14:10:29.661 DEBUG:teuthology.nuke:shortname: smithi077 2024-05-06T14:10:29.662 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-06T14:10:29.686 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688245', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-06 13:16:29.780457', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJohsaqWYVHYNRlz7jBCzGjRhYWqxUS1GR2fYpLwh5zEhXg0vrmTTjWDoEhZ2EuajqCBbSUQ0dkrh7VBd8HVGj4='} 2024-05-06T14:10:29.690 DEBUG:teuthology.nuke:shortname: smithi170 2024-05-06T14:10:29.690 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-06T14:10:29.716 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-03_21:08:03-orch-squid-distro-default-smithi/7688245', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-06 13:16:29.779135', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDGW/aOJzVnJ8BaA7lVrBx7YOJewdcCurFHn9O6QQmS4wk7S61m/iZdGqVMkWBKU0u0DPwdrDlWuRNJ1JF/qd6M='} 2024-05-06T14:10:29.745 INFO:teuthology.orchestra.console.smithi077:Power off 2024-05-06T14:10:29.745 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-06T14:10:29.771 DEBUG:teuthology.orchestra.console.smithi077:power off output: Chassis Power Control: Down/Off 2024-05-06T14:10:29.771 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-06T14:10:29.786 INFO:teuthology.orchestra.console.smithi170:Power off 2024-05-06T14:10:29.787 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-06T14:10:29.799 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-05-06T14:10:29.811 DEBUG:teuthology.orchestra.console.smithi170:power off output: Chassis Power Control: Down/Off 2024-05-06T14:10:29.812 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-06T14:10:29.835 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2024-05-06T14:10:33.800 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-06T14:10:33.836 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-06T14:10:33.911 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-05-06T14:10:33.948 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2024-05-06T14:10:37.914 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-06T14:10:37.949 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-06T14:10:38.026 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off 2024-05-06T14:10:38.059 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is off 2024-05-06T14:10:38.128 INFO:teuthology.orchestra.console.smithi077:Power off completed 2024-05-06T14:10:38.160 INFO:teuthology.orchestra.console.smithi170:Power off completed 2024-05-06T14:10:38.265 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} duration: 2920.5464475154877 failure_reason: '"2024-05-06T13:44:23.366285+0000 mon.a (mon.0) 996 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-05-06T14:10:38.265 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-06T14:10:38.346 INFO:teuthology.run:FAIL